6 Module VI: Experimentation
Experimentation: Testing and Analysis
Elissa Ledoux; Mohammad Uddin; and Matthew Sheppard
ABET Student Outcomes
ENGR Student Outcome 6: an ability to develop and conduct appropriate experimentation, analyze and interpret data, and use engineering judgment to draw conclusions
ET Student Outcome 4: an ability to conduct standard tests, measurements, and experiments and to analyze and interpret the results to improve processes
Testing and analysis prove if a prototype or product meets the design constraints and lives up to its name. A finished build is not a finished product – one can’t know a design is good unless its tested! This module covers best practices for prototype testing: data collection and analysis of results.
VI.1) Data Collection and Testing
Information
How does an engineer know if a system works as it should? How does an engineer know if a system meets the design requirements? He must test it! Quantitative testing is done through an organized procedure of data collection and statistical analysis. These statistics help the engineer to make decisions, solve problems, and design (or redesign) products and processes. The first part of the testing process is data collection, explained in the slide deck below.
Data Collection and Analysis Protocol in Engineering slide deck
Examples of test procedure / data collection forms are below:
Activity
Write a procedure for testing one aspect of your system or subsystems, following the examples above. (Each team member should write one for testing a different subsystem.) The document should include:
- Name of test and system/subsystem
- Required equipment
- Required personnel
- Objective
- Safety procedures
- Step-by-step test procedure in checklist form
- Photo of testing setup if applicable
The level of detail should be such that any competent classmate, engineer, or technician could perform the test and collect the data according to protocol without your help.
VI.2) Results and Analysis
Information
Once the testing data is collected, it needs to be analyzed in an organized fashion. Ideally, the results prove that the prototype meets the project design constraints both qualitatively and quantitatively. Qualitative pertains to the quality of results; i.e., how well does the prototype work (excellently, okay, poorly, etc.) Quantitative numerically characterizes performance in terms of accuracy, repeatability, and anything else relevant. It measures and reports actual numbers in table or graph form, analyzes those statistically (average, standard deviation, etc.), and compares them to the calculated/predicted/goal values.
Different methods of statistical analysis are appropriate for different types and quantities of data. Common methods are:
- Two-sample hypothesis testing
- ANOVA
- Regression Analysis
This Statistical Analysis Methods slide deck explains the steps and examples to conduct each type of analysis.
After the data is collected and analyzed, it should be displayed in a manner that is easy for everyone involved to interpret. This includes engineers, managers, and customers. Charts and graphs are the most preferable method of visualization, but plots and tables can also be used depending on the type of data. This slide deck on Creating Data Summary and Data Visualization provides recommendations and best practices for data display.
If the prototype does not meet or surpass the design requirements, a new iteration of the design cycle is required. If performance falls just barely short, only minor revisions are necessary, so tweak the prototype to improve performance. If the prototype breaks during testing, or proves inaccurate, unreliable, or woefully short of the design requirements, major revisions are necessary, so go back to the drawing board, redesign, rebuild, and retest.
Below are examples of results and analysis for prototypes that neared or exceeded all design requirements:
- Testing and Analysis Example – Robotic Lawn Mower
- Testing and Analysis Example – Rubik’s Cube
- Testing and Analysis Example – Voice Ctrl Toolchest
Activity
After testing your prototype according to the testing procedures your team developed in the section VI.1 activity, analyze the results using appropriate methods as described above, and catalog them in the worksheet below. All design constraints can be evaluated qualitatively, and choose 3-5 aspects of your system’s performance to evaluate quantitatively as well.
Testing and Analysis Worksheet
VI.3) Time Studies
Information
A time study is a cycle time analysis done for a task in order to identify minimum possible times and improve the efficiency of the process. Each process can be broken down into sub-tasks, videoed, and timed. This is done for several repetitions (sometimes with different people or machines depending on what activities are involved in the process), and the times calculated for each sub-task. Then areas for improvement can be identified and corrected in order to optimize overall cycle time.
The slide deck Time Studies Slides can be used by instructors to provide an overview of time studies and a guided class activity.
This Time and Motion Studies video by Beginning Engineers explains the basics of conducting a time study:
Their next video, Time Study Examples, provides multiple examples of breaking down a process and conducting time studies:
Activity
Following the procedures explained above, perform a time study for your project. First, break your system’s process down into sub-tasks using the process flow diagram you developed in Ch. 2. Create a form for recording time data by trial and sub-task as shown in the examples above. Then test your system for multiple full cycles, while videoing. Re-watch the videos and note times for each sub-task, then identify areas to optimize. How can you make your system faster or more efficient? Can any superfluous motions be eliminated? Can any speeds be increased without sacrificing accuracy?
