Day 1: Organization
The first day kicked off with a discussion of the goals and technical details for the project. Stefanie and Jihoon presented some of the MIT and Brown material respectively.
Breakdown of Work:
- Figure out project structure. (How to rossify SLU.) (SLU is Spatial Language Understanding and is the name of our codebase.)
- Create new repo. Add everyone and check out the code.(Jon)
- Extract color information from block.(Jihoon)
- Adding color in visualization(Jihoon)
- Object Tracking(Jihoon)
- Testing Grabbing and placing (Stephen, Jihoon)
- Create videos of the PR2 paired with ROS logs.(Stephen, Sam)
- Extract object and robot trajectories from ROS logs export into XYZT sequence.(Stephen, Sam)
- Extract perceptual features from the logs.(Stephen, Sam)
- Create state/action model of the PR2 in the SLU codebase. (Stefanie, Arjun, Jon)
- Implement SLU ROS node (Stefanie, Jon)
- Upload videos to turk and get commands.(Stefanie, Arjun)
- Annotate data, train model, and run inference. (Stefanie, Arjun, Jihoon)
- Read the PR2 state and send it to SLU.(Jon, Sam, Stefanie)
- Send the SLU inference results to the PR2.(Stefanie, Sam, Jon)
- Demo! (Everyone!)
- Documentation and photograph(Jon)
- Front-end (Jon)
Day 2: Putting the Pieces Together
Tuesday Morning Status:
Rosbridge running. Begun SLU manager that reads robot pose as a transform, sends over rosbridge to slu, then slu makes a state.
State-Action space in SLU using the forklift cost function
Made a plan for messages for communication with pick and place manager.
Goal: End to end demo with a random cost function
Have the PR2 picking up blocks and putting them down in arbitrary positions and orientations. Have brought Sam and Stefanie up to date with the PR2.
Goal: Do some hacking to track blocks as they’re being pick and placed and improving pick and place. Finish scripts to make videos.
Wrote the Rosbridge python client implementation. Now working on end-to-end demo with Stefanie and in particular cobbling together an interface.
Working on stuff to help Stephen for robots for education – enclosure escape
3D visualisation modifications for arbitrary block placement
Goal: Continue both
Working with Stefanie on state action space; gettin ga map between what the PR2 things and what the SLU does. (A lot of MIT in house code).
Defined the state as a set of blocks (cachement area / placed block grid)
Actions defined: pick up and put down
Today: Working on connecting the SLU manager to the block builder manager using the rosbridge client.
Goal: Parsing SLU manager and converting it to a ROS message for sending over rosbridge.
Finished writing a new build manager in python. Defined the message types for passing between controller and SLU with Arjun. Worked with Stephen on block picking.
Currently: Debugging block placement.
Worked on zzz’s
Day 3: Initial System Working
At the start of day 3, we have a first pass of an end-to-end system. The system uses SLU, passing messages through a rosbridge client on the SLU side, to the ROS system running on the robot. Another web browser rosbridge client is used to control the overall system. Check out the video to see it in action:
Day 3 Status Update:
Rosbridge bug fixed, end-to-end demo complete, now just need to correctly complete cost function.
Goal: Colours correctly in colour manager, reuse same code for rosbag state extraction and live state extraction
Did end-to-end web stuff, voice stuff, web interface stuff.
Goal: assist Jihoon with problems, act as curator on previous stuff
Working to get grab block to work as robustly as possible. Now quite robust. Now ready for videos for MTurk
Helped get the state extraction to extract state and poses from rosbags offline for log postprocessing.
Goal: synchronize messages and message names
Day 4: End-to-end system
Summary thus far, in the words of Stefanie: “We have an entire system, we have the ability to make videos. It would be good to make some videos with logs and hand annotate some commands this morning, for process’s sake. Then, we should make an end-to-end demonstration of the whole system.”
Took jihoon’s video and uploaded it to turk, and got 10 commands back from turkers. A few bugs in the log parser extractor which were fixed. Can now read a rosbag and draw the trajectory of the robot in SLU.
Made stub training targets for SLU so all that’s needed is YAML files and then it can just go.
Goals: Get end-to-end system going. 5 videos, 5 logs.
Working on picking and placing and is confident that it’s robust as it needs to be. Then started making videos with Sam.
Worked with stephen to get it working. Did block colour stuff. Writing instructions.
Working on the tracking server with Stephen
Little bit of help for Jihoon, looked into ripping out the wviz visualization
End to end system
5 videos, 5 logs, 2 commands
Demo video of end-to-end system.
Run everything from MIT machine and Brown machine
Next steps: repository, 25 videos
The video below shows a demonstration of the most basic functionality of the system. With a few more demonstrations (from MTurk) and a wider variety of commands, we will see some very interesting behaviour. Watch this space!