Forms that currently work with queXF
All forms created using queXML should work with queXF (including those exported by Limesurvey and queXS.
If you have a customised form, queXF requires the following to operate:
- A unique interleaved 2 of 5 or Codabar (from 1.4.0) barcode on each page
- Please note that the barcode must be unique for each page across all forms in the queXF system.
- By default, the barcode must be 8 digits long. Otherwise you need to change the BARCODE_LENGTH_PID directive in the config file
- A vertical and horizontal line in each corner of the page (for alignment)
- Boxes to be filled in should be square or rectangular (not circular)
Please note that the author has not tested a form that was not created in queXML, therefore can make no guarantees that it will work (Although it should). If you have any forms that you have trouble with (that comply with the 3 points above), please send in an example for testing.
Using queXF:
queXF has only been tested on a LAMP system (Linux, Apache, MySQL and PHP) and this guide assumes the reader is using a similar system.
Scanning forms
Currently, forms will be best able to be read in if they are scanned in with the following options on your scanner:
1. Monochrome (1 bit)
2. 300 DPI
Administrative Tasks:
These tasks are accessible from the /admin directory of the package (eg: /quexf/admin)
Import a form: New for queXF 1.10.0
Use the test_original.pdf in the /doc directory and the test_original.xml as the banding XML file when testing.
Import a form
If you are importing a new form with a banding XML file included, you don't need to scan in the original. Just import the original PDF generated by queXMLPDF and the included banding XML file.
It is best to import a form that is scanned in the same way as the completed forms will be scanned in. Therefore if you have used queXML to produce a form like this example, scan it using your scan software to a PDF file like this example.
Go to the page: Import a new form from a PDF file (new.php) and select the pdf file to import. (An example is test_blank.pdf in the doc directory). Enter a description such as 'test questionnaire'.
You should see a result such as:
BARCODE: 00000100
BARCODE: 00000102
BARCODE: 00000103
SUCCESSFULLY INSERTED NEW QUESTIONNAIRE
If you try to import the same form again, or another form that has a page with the same barcode as a previous form, you will get an error like this:
BARCODE: 00000100
BARCODE: 00000102
BARCODE: 00000103
FAILED to insert new questionnaire. Could have conflicting page id's
This is because queXF identifies each page by the barcode, therefore there cannot be a duplicate.
queXF Page setup (from 1.11.0)
queXF needs to know what size your scanned pages are, and the location of the corner markings on the page. These settings are stored in config.default.php by default, but different printing and scanning processes may lead to these settings needing to be tweaked.
Once the form has been imported, you will be linked to the page setup for the form. Choose a page to work from and a zoom level that is comfortable. Then you can drag, drop and resize the corner boxes to overlay them as they are on your form.
If the boxes appear to be in the right position, you do not need to move or resize them. If they do not appear to cover the expected area of your scanned documents - enlarge and/or move the boxes.
Blue lines should appear to overlay over the corner edges. This means that queXF was able to detect the corner lines on your form. If they do not appear in the right place, keep moving or resizing the overlay boxes until they do.
Band a form: New for queXF 1.10.0
If you have imported a PDF file with a banding XML - you don't need to band! You can skip this step. Otherwise you can look over the banding to confirm that it has imported correctly
Band a form
'Banding' a form is the act of telling queXF where the fields are that respondents will fill in. queXF is able to detect single choice and multiple choice check boxes, text and number boxes and also VAS (Visual Analog Scale) question types.
Banding a form works in two stages, firstly identifying the fields, then assigning the fields a name and type. Identifying the field is done by clicking once on the form at the upper left corner of the field, outside the field boxes, then clicking for a second time in the bottom right corner of the field, outside the field boxes.
queXF will then identify the boxes within the field and display them in red. The second task is to click on the boxes to bring up a window to assign the field name and type, or delete the field if it has been detected unsatisfactorily.
Banding using the interactive system is done by clicking and dragging around a field of boxes, then clicking within the selected result. The detected boxes should then be visible on the form. Right clicking on the boxes will bring up a menu allowing you to select what type of box field it is.
Once the form has been banded, go back to the main banding page and select 'reorder variables'. This will automatically order the form from top to bottom, left to right so that when the form is verified, it will go through in a logical order
Add users
Users must be added to the system that match the underlying authentication system from the add users page.
Assign forms to users
Once the users have been added, assign the users you want to work on each form
Intelligent Character Recognition (ICR) - new in 1.12.0
For an overview of the ICR system in queXF, see the queXF Intelligent Character Recognition page
Train ICR
This feature allows you to train the ICR system to recognise characters based on your own already verified forms in queXF.
Firstly, select a questionnaire to draw characters from, then use the checkboxes to select which verifiers you want to include characters from in the training process and click on "Continue training" (selecting none will include all).
A list of characters and the number of instances of them will appear. To train a batch of characters, use the checkboxes to select which characters you want to appear in training, then click on "Start training process in the background". This will create a knowledge base from the selected characters in the background. It will take approximately 1 second to train each character.
To manually train characters or to see what the system will be automatically training - click on the "Manual train" link. This will bring up a page displaying each character in a green highlighted box. Click on any box you wish NOT to train (it will turn red) - then click on the "Train" button to train the system for these characters.
The paper the ICR process in queXF is based on suggests that you will need approximately 400 instances of each character to achieve good recognition.
Monitor ICR training process
When you are training characters in the background - you can use this page to view the progress of the training.
Import and Export ICR Knowledge Base (KB)
This form lists all of your knowledge bases and allows you to click on them to export them as XML files - useful for backing up the knowledge base or for sharing it.
Use the upload feature to add a knowledge base from an XML file provided to you.
Assign ICR KB to questionnaire
Use this form before importing forms for a questionnaire if you wish the system to apply ICR on the characters for that questionnaire.
Select the questionnaire, then use the checkboxes to assign knowledge bases to either the "Text" or "Number" fields that have been banded for your questionnaire. This allows you to create specific knowledge bases for the different types of character entry (text or number) and to choose which knowledge bases apply for that field type for that questionnaire.
If no knowledge base is assigned to a box type, then no ICR will be performed for that questionnaire. Also please note that ICR will only apply for all forms imported AFTER the assignment takes place.
Import a directory of forms
Once the form has been banded, go to the import a directory page to import PDF forms to the system. Make sure you know the full path to where the files are stored on the web server, and that they are read accessible by the web server process. There are 2 filled example forms in the 'filled' directory of the doc directory.
If queXF is stored on the server here: /var/www/quexf then enter: /var/www/quexf/doc/filled to process the forms in the filled directory.
Please note that processing forms does take some time so please be patient.
Pages missing from scan
Where the VERIFY_WITH_MISSING_PAGES config directive is set to false (default), scanned forms that have a page missing will not be assigned to operators for verification. This page will list those forms. If you set VERIFY_WITH_MISSING_PAGES to true, then the forms will not appear on this list and instead will be assigned to verifiers with the missing pages appearing blank.
If the form has a page missing, but also undetected pages, the link will point to the "Handle undetected pages" form (see below for more details). If there are no other candidate pages, then the original scan must have a page missing compared with the template. In that case, you can click on the "Delete form and allow importing again" link to delete the form, and to allow queXF to import the same one again. You should re-scan this form and place it in the import directory before clicking this link.
Handle undetected pages
The undetected pages form allows you to manually tell queXF what page has been scanned where queXF fails to detect it automatically. queXF sometimes fails to detect a page if the barcode has been modified or scanned in poorly.
The page on the left is the 'undetected page' and the page on the right is the page that queXF thinks is a match. If queXF does not produce any matches, then the page is probably blank, or has been scanned in twice for that form. In that case, press the 'delete' link to delete the page from the system. If the page on the left has useful information on it, queXF should produce a list of possible pages on the right. If the page matches, press the 'accept' link. If the page does not match, there may be more available using the page numbers on the top of the right hand image. Once you have found the correct one, press the 'accept' link.
Output data
The output data form is used to export data from queXF. It will only export the data that a verifier has confirmed, not what has been automatically detected by the system.
Output unverified data
The output unverified data form is used to export data from queXF. It will only export the data that has been automatically detected by the system using OMR or ICR of all forms that have been imported, whether or not they have been verified.
Display Progress
The progress page displays the progress of form verification.
Client management
Adding clients to the system is similar to adding operators (you will require an underlying authentication system for this to work). Once a client has been added to the system, you can assign forms to them using the "Assign clients to forms" tool.
If you expose the /client subdirectory to the internet, with the authentication system active - clients will be able to review their forms page by page by entering the form id. As the form id is output by default in the data file - the client can view the scanned form that relates to their data.
Integration with queXS (from 1.13.0)
If a paper form is generated from the queXML output from queXS, it is possible for queXF to upload each verified paper form back to queXS (using XML RPC), so that there is a single dataset for analysis.
Requirements:
*queXS 1.5.2 or above
*Form in queXF generated from the queXML output of queXS
Once the form has been imported to queXF (including the Banding XML file) click on the administrative menu "queXS and Limesurvey integration" and select the form.
In the field "RPC Server URL" enter in the address of the RPC file. For example:
http://user:password@localhost/quexs/include/limesurvey/admin/remotecont...
Where user is the username, password is the password of your queXS directory authentication, and localhost is the server.
Currently the username and password fields can be left blank, and the Limesurvey Survey ID field should have the correct default value loaded.
Click on "Update settings" to save the settings.
If there are forms listed - these will be the ones that have already been verified but have not been uploaded to the RPC Server. Click on upload to upload them individually.
Operator Tasks:
Verification
Verification is where an operator runs through each question in a questionnaire to confirm that queXF has correctly guessed what has been written on the form. If there is
a discrepancy, the operator can correct it.
Each question is run through. The question that the operator is working on will be highlighted orange. The guessed answer will be shown in green. The operator may use the mouse to select/deselect an answer by clicking on the appropriate box, or may use the keyboard as shown below:
To "Zoom in" on the page using a pop- up window press the F2 key.
Keys for single choice and multiple choice question types:
- Enter: Accept the current answer and move to the next question
- Ctrl: Move back to the previous question
- 1-9: Select/Deselect the 1st,2nd...9th choice
- left, up: Move the cursor to the next choice left or up
- right, down: Move the cursor to the next choice right or down
- Space bar: Select/Deselect the current box
Keys for text and number question types:
- Enter: Accept the current answer
- Ctrl: Move back to the previous question
- Backspace: Move back a box
- Space bar: Leave the next box blank