Data Plotting

We are now able to plot the human model using only 7 points of data down from 73 points. Thanks to Kester's Flash client.

Exception Handling

I have edited the backend programme to handle exceptions. More common errors are that the pictures are too blur for the face or the pattern card to be detected. Other errors are that the user did not take a full picture, where his legs aren't captured in the picture.

Weekly update 11

Done:

• Integrated back end program. [A]
• Rewrote Data Point Module for more dynamicplotting. [A]
• Integration of client, server-side script and backend program. [K]
• Improved Interface and wrote temporary datapoint program. [K]

Final Lap:

• Plotting of the shirt mask. [K]
• Migrate system to zone account [K]
• Improve the visual aesthetic of models and shirt generated. [K]
• Collect the data point and body measurementaccurately. [A]
• Testing will also need to be conducted on thesystem and the users. [A]
• Complete and wrap up the final project. [Both]

Some update ...

Project integration

Currently, I have tested out the flow of data from client to server script to the backend program. Over the weekend, while waiting for the backend program to be up, I have written a temporary datapoint program to test out the script requested for datapoint. The communication between the client and server is successfully done and manage to generate data point from the 2 image submitted by user. I have yet to test out the actual program written by Alvin yet, still some bug that we need to figure out. We will be doing it later in the morning, hopefully can get it done before the presentation.

The temporary datapoint program can capture the upper body of the person. The program simply scan only the left half of the body. It scan the image line by line horizontally and vertically storing the datapoints in 2 arrays respectively. After which, the two arrays are combined. The arrays combine when both reach a common meeting point (the green point see below figure). The result looks weird especially at the body portion (too striaght), and the hand looks like alien. But well, this is just for testing purposes. Hope the final one will look better.

Illustration if line scanning:
Result of datapoint plotted out from a noise-free image after differencing:

Other update:

-Drawn logo to fill up the empty space on the left side of the interface. The logo area can slide up when user navigate into the dress-up area.
-Added profile update functionality.
-Presentation for tomorrow (edited from alvin's version)

Weekly update 10

Done :

See Integration of Server Programme
See Client & Server-side scripting update

Todo:

Complete prototype 2

Integration of Server Programme

I was able to integrate all that is required in the server side application together in one big programme. It detects all the necessary part and objects and is able to retreive the relevant pictures from the different folders and save them into different folders too. The basic input of the programme is purely the username of the user, and all will be done by my programme.

I am still trying to get a better plot for our data point plotting as it is not generic enough (hope I could get it done asap).

Another problem was that we were unable to execute the programme through the PHP command exec(). Hope someone can help!

Client & Server-side scripting update

Some update about the current state of our system. The client side is quite fully functional, on top of what has been built last week, the login and register and uploading and photo taking functions, this version allows the user to send the image of herself in the new outfit to her friend via email.

The integration of backend image processing and communication is still in progress. Just recieved the datapoint program (which detects 5 critical datapoints) written by Alvin last night and I'm now still trying to figure out the exec() function in PHP, it is still unable to run correctly. We will also have to change the plotting method use to generate the model.

_________________________________________________

Below are the more detailed description of the updated version:


1) Converting the the image of the dress-up model into picture for email


In order to convert the image in swf to jpeg, we need to convert the the movieclip into bitmapdata. Because the image created after the user has dress up herself is actually made up of many movieclips (the model, the head, the shirt and pants mask), we cannot simply convert/combine them into a single movieclip. A single movieclip is required so that we can use it to generate a bitmapData which can then be converted into jpeg using BitmapExporter (similiar to the one use for webcam pic function).

To resolve this problem, all the movieclips on the model were converted into bitmapdata. I have also written an pixel by pixel analysis in actionscript to read in the bitmapdata of each movieclip using bitmapdata's getPixel method. An invisible 1x1 pixel movieclip is created, which will scan through the whole area where the model is standing and help to detect which movieclip is on 'top', thus helps to return the correct RGB values. The RGB values are written into an 2d arrray which is then use to draw a new single bitmapData.


2) Email functionality

User can now send the image of herself and request some comment from her friends. I have tested out it localhost using ArgoSoft Mail server, below is the screen shot how it looks:

3) User can download of pattern card for printing. Below is the new patterncard with guided dotted lines:

4) Written scripts to handle the images uploaded, images taken via webcam and images created by user after dressing up in iApparel. The script is to convert the images with arbitrary name into correct naming convention.

5) New look for iApparel. Designed to fit into the Window Vista environment.

Weekly update 9

Completed:

- Client and server-side scripting. System is now able to register new user, login account, upload of user photo and taking picture via webcam. [K]
- Converted the BitmapData from flash to jpeg format with PHP. [K]
- Attached face cut-out on model. [K]
- Improvement made to the image differencing. [A]

Todo:

- Complete prototype 2 for next week product demo. For details, please refer to the prototype 2 report which will be uploaded soon.

Improvements to Image Differencing

Several improvements were made to the Image Differencing Programme. It checks for individual channel of a pixel instead of a pixel as a whole. It is more accurate than the previous function.

Client & Server-side scripting update

In this version, the functions avaliable in the client side application are login menu, registration form, photo uploading and photo taking via webcam. The front end is mainly flash, while the backend uses PHP and the database server used is MySQL. Currently, the application runs on localhost using an open source cross platform web server called XAMPP. Below are some screenshots of the current version.

Photo taking via webcam update

As discussed last week, the snapshot captured captured in flash is in BitmapData. In order to convert this data to jpeg, I have made use of the BitmapExporter class obtained from quasimondo to compress the image data, which is then send to the server-side for processing and saving it as jpeg image. The server-side scripting language used is PHP as it has a GD library that is capable of creating image dynamically.

Weekly update 8

Done:

• Masking the shirt and pants area [K]
  
• Image texture loaded dynamically [K]


• Captured webcam image into bitmap data [K]


• Researched about image mesh warping [K]


• Image differencing - capture actual pixel color of the human [A]


• Get the actual dimension of the pattern card [A]

Todo (For week 9 & 10)

• Masks generated with critical points [K]
  
• Attach face into the generated model [K]
  
• Build client side [K]
  
  • Gather more clothes textures
  
  
  • Send sms data function
  
  
• CGI scripting [K]


• Build server side [A]
  
  • Sms & email function
  
  
  • Send data points to cgi script
  
  
• Integrating the system [A]

*See calendar for detailed description

Photo taking via webcam

A simple photo taking function was done with the help of this tutorial. The code were modified to add the countdown timer functionality. Currently, the image captured is in bitmapData format. The next version will need to convert the bitmapData into jpeg format for upload into server. Also, we will need to look into improving the quality of the image captured.

Clothes area masking

In this version, the masking of the shirt and pants area has been done. Some cloth textures obtained from mediostream and mayang were loaded dynamically into the flash. Currently, the masks were drawn out manually, the next version will draw the mask according to some critical points, so as to support different type of apparel such as blouse.

Pattern Card Detection and Analysis

After a face detection, we will also need to do the pattern card detection as well as analysis. The pattern card detection, is somewhat similar to a face detection. It requires numerous trainings where many samples are being fed into the training programme. This will result in an XML document, which is the main document for the recognition part. This vigorous training has been done by Kester.

This detection programme, uses similar algorithm like the face detection programme, in which a different XML document was used.

Next was the analysis portion. After the detection is completed, a smaller image will be generated by the programme will be used for the analysis. The smaller image is converted to a gray scale image, in another words, only black and white pixels present. This make the coding somewhat easier, since I only have to deal with 2 kinds of pixels.

The analysis went quite well and I had easily find the size of the pattern in pixels. This information is useful in calculating all real measurements such as the height and the shoulder-width of the user.

This is done using the line scanning technique discussed before.

Image Differencing Completed

I am now able to get a relative good difference in the two pictures. Supposed we had the 2 images below:

Picture 1:

Picture 2:

Using the OpenCV built-in function, which is a direct subtraction of array values. It is not of any value to us as it causes very funny colour to the picture. (see photo below)I rewrite the whole function which subtracts the array values, but maintain the picture's original colour. That will be a picture useful to us then. Below is the photo that uses my function. It will look better if a higher resolution photo would be used. The background could be any colour the user chooses and even another background by changing some numbers for the BGR of the pixel.

Weekly Update 7

Completed:

1) A test out of Model Generation using data points in Flash Interface. [K]

2) Extraction of face from picture/camera using Face Detection. [A]

3) Survey on Project Design. [A]

TODO:

1) To speed up the prototype 2 for presentation in Science Centre.

2) See Calendar for latest update.

Model Generation progress

A simple version of model generation was done according to the body's "data points". In this version, it is more like plotting out the silhouette of the human. A dummy set of data was used for plotting and they are connected using flash's lineTo function. The model's skin is filled with skin's samples from 3dmd and the filling is done with beginBitmapFill method.

Subsequent will look into masking the clothes area of the model and improving the visual aesthetic of the model.

Face Detection

After analysing the OpenCV, I am able to edit the face detection sample code to use in our project. I was able to extract the face from a picture and save it as another picture. It works well for some pictures I have tested, even for virtual models.

The process is fully automated and purely written using OpenCV Library. The program will read an image each time, and write out an image with a file name using the userid of that user. Therefore it will save as a different image each time.

Sample picture tested:

Result:

Weekly update 6

COMPLETED:

1) Accurate pattern card detection and obtaining the dimension of the mark out rectangle for comparision with silhoutte. (See Object Detection Progress)[K]

2) Detailed line scanning of image in Java & differentiating color of pixels. (See Line Scanning)[A]

3) Paper prototyping on flash, able to click on sample clothes and map on the user in the picture given. [K]

4) Powerpoint presentation for prototype 1. Analyses on the current interface used for exsting market, the design problem of initial concept and possible solutions.[A][K]

TODO:

1) See Milestone update in Prototype 1 Presentation

2) Pipeline testing

3) Make nescessary change to project according prototype 1 presentation feedback

Line Scan 2nd Update

My line scanning program searches for prominent black pixels (at least a set number of pixels), and would not take in random black pixels as there may be a lot of noises in the actual test out.

It detect a non-black pixel by executing

"

return (colour != BLACK) ? true : false;

"

where BLACK is an integer holding the super big number, 256*256*256.

DEMO

Bad Points to note:
The program will be doing a for loop, extracting all the RGB value of the pixels' colour. This algorithm is a O(n²) algorithm, and takes a very long time to execute. We will be working on the algorithm again to decrease the time and execution speed to at most a O(n) algorithm. This reason is that if the picture we were having is very big, the user may need to wait a very long time before he can get a result. It may not be very good for commercial uses.

Prototype 1 Presentation Draft 1

Aim of this presentation is to :
1) Update on the progress of our project
2) Serve as a paper prototype to get a feel of how our project will look like base on our initial idea.
3) Analyse the pros & cons of existing system in market
4) Possible solutions
5) & ...

Line Scanning

Using the image differencing programme tested out from Kester, I expliot the use of its properties that the image will have a black background after it was differenced out.

Using PixelGrabber from the Java Image API, I was able to get each and every pixel RGB colour from one row of pixels from the image. By running a for loop on each row, I am able to identify which pixel is black or non black.

Using the PixelGrabber, the row of pixels' colour will be stored inside an int array, a black colour will have a value of -16777216, which is equivalent to the value of each RGB colour, 256 for each primary colour. We will use this colour or value to find out the position of each Critical Point as stated in our programme.

Object Detection Progress

The object detection of our "I" pattern card has been quite successful. Below are some pictures and details about the training process:

15 Sept 07
1) 115 pictures were collected as positive samples. For each picture, an average of 4-5 were mark out for sampling using haartraining's objectmarker.exe. In total, 468 samples were created. 1000 negatives samples (with arbitary background) were used.

Some variables such as i) background color and lighting ii) design and color of what user is wearing iii) angle of the pattern card iv) no. of pattern card v) posture vi) distance from camera etc (sit or stand) were all taken into consideration.

Samples positive pictures for training:


Different color pattern cards used:



2) Training of data were conducted twice. First was conducted on 15 Sept, results of test is not accurate, but managed to detect the "I" pattern. In this training session, only 30 negatives image were used. In the second training conduct on 16 Sept, 1000 negatives were used instead. The training took around 13 hours to complete (16 Stages of training)! Results are quite satisfatory with minor problem (i.e detect extra "I")

Results of 1st training:


Results of second training :


A few test sample detect "extra" stuff:

Some possible solutions to address this: i) training more negatives ii) Let user decide where the "I" is exactly (See Virtual Makeover where user locate exactly where the location of their eye is)

3) Also edited the testing application (performance.cpp) to extract the X, Y-Coordinates, the width and height of the detected I enclosing box. i.e we are able to obtain ratio of real life object.

iApparel

Introducing our project name:

iApparel

Project Name

We would need to think of a name for our project.

Suggestions welcomed.

Calendar Updated.

I am thinking that our project is to be done by term break for a more refined usability test. More should be done on the integration and testing.

Project Milestone

Finally got our calendar online. Thanks to Google Calendar for making my life easy to post events on the calendar.

Proposed flow of work till term break:

1) We feel that a prototype would be necessary as our product is to be used on the web, which is viewed by many people. We would want them to have a good user experience on our product, therefore a user testing is necessary if we were to proceed further. A paper prototype would be completed by 18th Sept, or a semi-working prototype to work then. I will be setting up a mock-up terminal on 20th Sept at SoC COM1 to do the user testing. (Hope to see you there!)

2) We are training our pattern cards at the moment, with different backgrounds and different angles. We will try to aim for a perfect pattern recognition for our pattern card.

3) I will be also developing on Line Scanning on images, getting critical points of the human anatomy, eg. the tip of the head, neck, waist etc... This will be done on the image after the Image Differencing done by Kester.

4) Face detection is another vital part of our project. It is the most crucial part as it identifies the user and gives them a more customised experience when they are using our system. This had to be done precisely. Our test programme had already identified the faces quite nicely, where our work will be to extract the face out, saving it as a file inside the server.

~by qinjie

Weekly update 5

COMPLETED:

1) Tried out the face detection application (week 4)

2) Able to extract silhouette of the person by image differencing (week 4)

3) Experimented on the haartraining application of openCV for object detection the "I" sign used for our pattern card. Able to train sample pictures and successful convert trained data into xml format using cascade2xml.exe. Testing result still not satisfactory. See below.

TODO:

1) Thoroughly train actual pictures of different size/color/distance away from the a few users. Ensure the program can accurately detect the pattern.

2) Do comparision of the pattern card and the silhouette of the person get estimate of the body measurement.


-------------------------------------------------------------------------------------

Some of pictures use for training:



No. of samples trained: 42
No. of stages in training: 52

Test result:

About project update

We will be using this blog to do our project update as well. This is our first draft of project doc. Also there will be a post with the topic "Weekly update X" (X is the week number) containing 2 items:

1) COMPLETED - a description of what has been completed in week X.
2) TODO - things/tasks to be completed next week (X+1).

Ideas Welcomed!

Anyone with fantastic ideas out there? Clothes aren't just fantastic ideas after all. We welcome more ideas for interactivity, perhaps something unusual, with something usual at the same time.

Project Mates Wanted!

The E-sampling project currently has only 2 members, Kester and I. I am reaching out to anyone who are interested in this project! Please email me if you wanted to become one of our project mate...

Start

I have decided to create a blog for this CS3342 Project. E-Sampling will be the theme of our project.

More to come soon...