Abstract
Noticeable improvements were achieved in the method for quality evaluation of formed paperboard containers. The method now allows for in situ evaluation of unavoidable wrinkle structures along the sealing rim of formed containers. An image of the sealing rim was provided. In this image, the contour of the sample was detected. The contour line was then offset to the inside of the sample, so that the new line was on the sealing rim, regardless of the original contour geometry. Along this offset contour line, the wrinkle structure was evaluated by using a previously described cross-correlation-based method. The repeatability and accuracy of the method were validated by comparing the detection results with the results from thorough human examiners. Furthermore, an approach to find the optimum settings for the wrinkle detection program is described and an outlook on implications for industrial adaptation of this method is given.
Download PDF
Full Article
Geometry-invariant Wrinkle Detection in Sealing Rims of Paperboard Containers
Tobias Müller,a,* Alexander Lenske,a Christophe Barbier,b Marek Hauptmann,a and Jens-Peter Majschak a
Noticeable improvements were achieved in the method for quality evaluation of formed paperboard containers. The method now allows for in situ evaluation of unavoidable wrinkle structures along the sealing rim of formed containers. An image of the sealing rim was provided. In this image, the contour of the sample was detected. The contour line was then offset to the inside of the sample, so that the new line was on the sealing rim, regardless of the original contour geometry. Along this offset contour line, the wrinkle structure was evaluated by using a previously described cross-correlation-based method. The repeatability and accuracy of the method were validated by comparing the detection results with the results from thorough human examiners. Furthermore, an approach to find the optimum settings for the wrinkle detection program is described and an outlook on implications for industrial adaptation of this method is given.
Keywords: Paperboard; Deep drawing; Forming; Quality evaluation; Image processing; Packaging
Contact information: a: Institute of Natural Materials Technology, Chair of Processing Machines and Technology, Technische Universität Dresden, Dresden, Germany; b: BillerudKorsnäs AB, Gruvön Mill, Sweden; *Corresponding author: tobias.mueller1@tu-dresden.de
INTRODUCTION
Through the recent works of Hauptmann and Majschak (2011), Hauptmann et al. (2014), Hauptmann et al. (2016), Hauptmann (2017), and Vishtal (2015), the geometric boundaries of press forming and deep-drawing of paperboard have been advanced from simple round cups and oval trays to complex packaging containers with concave sections and multiple geometric features. Some examples from these shapes are shown in Fig. 1.
Fig. 1. (a) Concave sections (Hauptmann et al. (2014)), (b) sharp edge radius and different radii (Hauptmann et al. (2016)), (c) concave/convex transition (Hauptmann and Majschak (2011)), (d) complex geometry (Hauptmann (2017)), and (e) with sealing rim
For an industrial application of the technology, robust process and quality control is important. A forming, filling, and sealing machine for paperboard packaging needs to reach an output of ca. 60 parts per minute to be economically sensible. Machine downtimes and quality rejects are to be minimized. A robust and inline quality evaluation is the prerequisite for such an application.
For quality evaluation and assessment of the forming process, the wrinkles in the sealing rim of the paperboard container are examined. Broad wrinkles are optically displeasing and can prevent leakproof sealing of the container (Leminen et al. 2015). This also reduces the quality output of a forming machine.
Previously, the detection and evaluation of wrinkle structures in the wall section or on the sealing rim have depended on the specific geometry and measuring setup. Müller et al. (2017) used laser topography that relies on scanning the container along a predetermined contour. The setup of Wallmeier et al. (2015) prohibited wrinkle detection on non-round geometries, and Leminen et al. (2016) required predefined regions of interest (ROIs) on the sealing rim for evaluation to work. The method described by Müller et al. (2018) was able to evaluate round and elliptical shapes, but it was limited to the observation of the wall section in two distinct ROIs.
The current study is a continuation of the previous work and attempted to comprehensively advance the basic method detailed by Müller et al. (2018). Industry requirements for a fast inline evaluation of different geometries were addressed. The goal of evaluation one sample per second was reached and meets the industry requirement that every sample can be scanned at a machine output of 60 pieces per minute. The advanced method and setup allowed for an expeditious and accurate quality evaluation of the sealing rim of formed paperboard containers. The entire sealing rim was evaluated in one step. The method was not limited to a single geometry, but was applied to a broad range of sealing rim geometries.
EXPERIMENTAL
Method and Setup for Image Generation
The evaluation chamber (Fig. 2) was comprised of a semitransparent, opaque pane, on which the sample was placed. The pane acted as a diffusor and was backlit by an LED-matrix. Above the opaque pane, a camera was placed, so that the entire size of the sealing rim of the sample could be photographed in one image.
Experiments with alternative lighting methods showed that only backlighting was robust against distortions (i.e., waviness and warping) in the sealing rim and could show the wrinkle structures with enough contrast to be discerned from the material structure itself. Tanninen et al. (2017) described a setup with backlit press forming samples as well.
Figure 3a shows an image of a sample obtained with darkfield lighting. In darkfield lighting, the light-source is positioned at a shallow angle towards the illumination plane. Generally, darkfield lighting is suitable to produce a high contrast on surface structures. It can be seen that the curvature and waviness of the sealing rim led to a strong shadowing effect on the sealing rim that prohibited sensible wrinkle detection. The image in Fig. 3b was acquired under a dome light, which granted a diffused and even light source from a half-sphere dome above the sample. Still, the sample was not lit homogenously along the edge of the sample and the contrast was not sufficient. Figure 3c shows the resulting image of a backlit sample. The wrinkle structures in the sealing rim can be seen with a high contrast, which was beneficial for the following wrinkle detection.
Fig. 2. Setup for image acquisition
Fig. 3. (a) Darkfield lighting, (b) dome-light, and (c) backlighting
The algorithm for wrinkle detection in the sealing rim was programmed in Python (Version 2.7.10.0, P, Beaverton, OR, USA), utilizing the OpenCV computer vision library (Version 2.4.11, opencv.org). The images for the wrinkle evaluation were loaded from a user-specified folder. The folder can contain an arbitrary number of images from many samples. Wrinkle evaluation was then performed in two steps: image preprocessing and wrinkle detection.
Image preprocessing
The original image was converted to an 8-bit grayscale image. This ensured faster processing times, as no information was needed for wrinkle detection because it was encoded in the RGB-channels. The edges of the image were cropped so that the sample was the only object in the image with a clear contour (Fig. 4a).
The image was then opened to remove noise and disturbances. The erosion operation was followed by a dilation operation as described by Haralick et al. (1987). The image was scanned for contours with the OpenCV function findContours, which utilized a contour detection method described by Suzuki and Abe (1985). The findContours function used a gray value-threshold to detect contours and listed all of the contours found in the image, ordered by size in pixels. The threshold was initially set to a gray value of 80, which was adjusted incrementally for stable contour detection. Of the contours, only the contour with the greatest length was kept and the rest were discarded. Contour detection could be assisted with a heuristic function regarding the expected contour length range. The length of the detected contour was measured and compared using the heuristic function. If the contour was within the range, it was used for further wrinkle detection. If the contour was outside the admissible range, the threshold value was increased and contour detection was started anew until an admissible contour was noted (Fig. 4b). This loop ensured that only the contour of the sample was detected and that contour detection was robust against the illumination variation between individual images.
Fig. 4. (a) Original image, (b) contour detection failed, and (c) preprocessed image with contour (red line)
Every pixel that was not enclosed by the contour was set to black (gray value = 0). Those pixels were outside of the sample and not of interest for detection of wrinkle structures. This image mask was then merged with the original, cropped image, which resulted in the final preprocessed image (Fig. 4c).
Wrinkle detection
The Cartesian coordinates (X, Y) of the contour line were transformed to polar coordinates (φ, ρ) and sorted by ascending angle φ. The coordinate transformation ensured that a continuous contour vector with unique φ-values was generated. To decrease the computing time, only every tenth (φ, ρ) coordinate pair was kept. The radial component ρ was decreased by a factor of 0.95 so that the contour line was on the sealing rim instead of on the outer edge of the sample. Finally, the coordinates were transformed back to Cartesian coordinates (X, Y) so that the program could identify each coordinate pair with a pixel (px, py) in the image. However, through the transformation and ordering of the coordinate pairs, now the (X, Y) coordinates were ordered so that an incremental sweep of the coordinate pairs in the contour vector was mapped to a continual sweep of the actual contour in the image.
As proposed by Müller et al. (2018), a template of an ideal wrinkle with changing grayscale values (GV(x)) was built symmetrically to the middle of the image on the X-axis (x0), according to Eq. 1, with the sharpness (sn) scaled to the size of the template itself. Figure 5 shows a visualization of the resulting template.
(1)
Fig. 5. Template of an ideal wrinkle (Müller et al. 2018)
The template was moved along the offset contour line. For each step, the cross-correlation of the template and the section of the image on which the template was currently overlaid was calculated and saved for later, along with the current coordinates and the rotation of the template with regards to the center point of the contour line. The template was then moved to the next step, rotated by the current φ, and the process was repeated as is indicated in Fig. 6. To save computing time, the size of the iteration step could be adjusted, but was at least half the size of the template so that the same wrinkle structure was not detected multiple times.
Fig. 6. Sweeping of the template along the offset contour line
At each point, the template cross-correlation and overlapping image section were calculated and written in a new column alongside the respective coordinate vector. After the cross-correlations for the entire contour were recorded, the array with the correlation values was compared with a correlation threshold. Any values that were smaller than the threshold were then set to zero and no wrinkles were marked as detected at this coordinate pair. The Euclidian distance between the coordinates of neighboring non-zero values was calculated and compared with a minimum distance. The minimum distance ensured that the same structure was not detected multiple times. Should there have been more than one non-zero correlation value within the minimum distance, then only the largest value was kept and all other values within the minimum distance were set to zero. This guaranteed that only the best fit was kept.
The remaining non-zero values in the correlation array were counted. This yielded the amount of detected wrinkle structures. Furthermore, the standard deviation of the Euclidian distances between neighboring points that had a non-zero correlation value was calculated. As described by Hauptmann (2013), the standard deviation of the wrinkle distances can be used as a characteristic for the uniformity of the wrinkle distribution.
The wrinkle structures were marked in the original image, which was saved separately (Fig. 7). The wrinkle data for this image was written to a database where it could be compared with the material and forming parameters of the original sample.
Fig. 7. Preprocessing, contour detection, and detected wrinkles
Invariance of the sample geometry and limitations
Because the contour detection and subsequent wrinkle detection were not explicitly developed for samples with a round geometry, it was possible to evaluate non-round geometries without changing the algorithm. Examples of square and oblong shapes are displayed in Fig. 8.
The ability to detect wrinkles on a broad range of sealing rim geometries enabled a flexible usage of the described detection method without the need for separate measurement setups. Additionally, the detected contour could be compared to an ideal contour so that variance in the geometry during the forming process could be evaluated.
The described method was limited to wrinkles in the sealing rim or a flange of the sample. Wrinkles in the wall section could not be detected. For detection of wrinkles in the wall section, alternative methods have been previously reported by Wallmeier et al. (2015), Müller et al. (2017), and Müller et al. (2018).
The applicability was limited by concave sections in the sample. As was previously discussed, the XY-coordinates of the contour needed to be sorted unambiguously by the respective φ angle from the center of the contour to establish a continuous contour. This was not possible for very concave sections, in which more than one pair of XY-coordinates could appear at the same φ angle (Fig. 9).
Fig. 8. (a) Wrinkle detection on square-like sealing rim geometry, and (b) wrinkle detection on oblong geometry
Furthermore, it has to be noted that while the angular steps in the algorithm were constant, the relevant steps on the actual contour were not constant. This was especially true for the samples that displayed great changes in their curvature along the contour of the rim (Fig. 9).
Hauptmann et al. (2014) discovered that the main challenge in forming concave geometries was the limited stretchability of paperboard and that wrinkles formed in different orientations in the concave sections. Those findings suggested that a different approach for wrinkle detection in concave sections might be necessary.
Fig. 9. Limitations of the geometry invariance in concave geometry sections
The described method required only a single image of the sealing rim and was able to detect the wrinkles on the sealing in less than a second using a standard laptop computer.
An optional infeed system for the imaging chamber (Fig. 10) enabled automatic sample processing. This is an important step for industrial applications, where samples need to be evaluated in situ. Prospectively, the inline evaluation can be used to optimize the sealing parameters according to the wrinkle distribution present.
Fig. 10. Autonomous module for automatic sample imaging and wrinkle detection
RESULTS AND DISCUSSION
The main parameters for the detection algorithm were the correlation threshold and size of the wrinkle template. To establish the optimum settings for these parameters, an empirical study was conducted. The wrinkles on 54 samples of different wrinkle quality levels were counted manually by two independent examiners. Then, the automatically determined, average wrinkle quantity for all of the samples was compared for different setting of the correlation thresholds. The correlation threshold setting that achieved the same number of average wrinkles as the human examiners ( = 38 wrinkles) was subsequently used for the automatic detection. Figure 11 shows that a correlation threshold of 0.75 was suitable.
With the determined threshold, the influence of changes in the morphology of the wrinkle template was evaluated stepwise. Figure 12 shows the results of the investigation of the influence of different template sizes. Again, the results were compared with the results from the average manual wrinkle count ( = 38 wrinkles) to acquire the optimum settings for the template size.
In this study, the optimum settings were a template size of 16 px × 16 px and a correlation threshold of 0.75. Because the empirical adaptation of the template only depended on the wrinkle size, it was expected that those settings are applicable for a broad range of materials and geometries, as long as the size of the wrinkles is within detection range.
Fig. 11. Detected wrinkles for different correlation thresholds (n = 54)
Fig. 12. Detected wrinkles for different template sizes (n = 54)
To validate the correct detection of wrinkle structures and distinguishing of the wrinkles, discolorations, and other imperfections on the sample, the same images presented to the detection algorithm were also inspected by a human examiner. In the first test, the resulting images with marked wrinkle structures (Fig. 13) were checked for false positive and false negative detections and omitted wrinkles.
Fig. 13. Sample with marked wrinkles for manual validation and false positive and false negative detections; the sample shown has more wrinkles than the average ( = 38 wrinkles)
In the second test, the human examiner was only presented the original images and instructed to mark and count the visible wrinkle structures. Figure 14 displays the wrinkle quantities detected with the algorithm and by the human examiner for the same samples.
Fig. 14. Comparison of the automatic detection and wrinkle detection by a human examiner (n = 40)
To investigate the repeatability of the detection results, the same images were evaluated several times. With the same images and settings for the contour detection and wrinkle template, the evaluation proved to be consistently repeatable without any deviation. Furthermore, 10 different images of the same sample were taken and evaluated. Figure 15 shows a boxplot of the variation from the mean wrinkle quantity for six separate samples. Each was evaluated 10 times with 10 individual images of the same sample.
Fig. 15. Boxplot of the variation in the wrinkle quantity when repeating the evaluation with 10 different images of the same sample
It is noteworthy that, while there was some variation in the wrinkle detection results when the program was confronted with different images of the same sample, those variations were comparatively small and allowed for a distinction between different wrinkle quantity levels.
Optionally, the wrinkle detection could be performed with slightly altered settings for the template size and sharpness. A comparison of the values and coordinates from multiple detection runs allowed for sorting of the wrinkles into different wrinkle classes. In a real sample, wrinkles of different sizes form; therefore, multiple detection runs with different settings can be used to accurately detect wrinkles of all sizes. With multiple detection runs, it was necessary to avoid double detection of the same wrinkle structure by comparing the scored correlation values for different templates and discarding the smaller values once more.
Variation in the size of the template used for wrinkle detection allowed for classification of the wrinkles into different classes of wrinkle sizes, as is displayed in Fig. 16. Using a variation of the template size improved the detection accuracy because a wide range of wrinkle sizes were detected.
Fig. 16. Detection of different sized wrinkles by using different sized templates (i.e., size 8 px × 8 px, 16 px × 16 px, and 24 px × 24 px)
In the future, the distribution of wrinkles in different classes can be used to optimize the forming parameters, so that the desirable wrinkle class becomes predominant.
CONCLUSIONS
- The described method improved upon the work of Müller et al. (2018) in a way that now allows for the evaluation of a broad range of geometries without the need to adjust or reprogram the wrinkle detection. The method was implemented in an autonomous module and is thus ready for industrial adaptation.
- The ability to evaluate the formed sample with a single image of the sealing rim in particular allows for fast wrinkle detection. The readily available wrinkle data prospectively enables the adjustment of further downstream process steps (i.e., sealing) for a specific wrinkle distribution on the sealing rim.
- The reliability of the improved method was high and tested thoroughly. The results of the method were validated by human examiners. Based on the manual examinations of the wrinkle distributions, a way to find sensible settings for the detection program was proposed and validated.
- The method offers a flexible and fast way to evaluate the quality of formed paperboard containers.
- The wrinkle evaluation worked repeatedly and reliably, regardless of the sample geometry and wrinkle sizes.
- The optimum parameters for the correlation threshold and template size were determined empirically.
- The results showed that the method is meeting the industrial requirements of evaluation speed, reliability and robustness against changing image quality. However, the correlation between wrinkle distribution on the sealing rim and actual sealability and gas tightness has not been studied in sufficient detail yet. The method described in the paper enables such a detailed study for future work.
ACKNOWLEDGMENTS
The authors gratefully acknowledge the support of the research group “Forming of Fiber-based Materials” at Technische Universität Dresden (Saxony, Germany) and BillerudKorsnäs (Solna, Sweden) for the financial support and use-cases for testing the developed method.
REFERENCES CITED
Haralick, R. M., Sternberg, S. R., and Zhuang, X. (1987). “Image analysis using mathematical morphology,” IEEE T. Pattern Anal. 9(4), 532-550.
DOI: 10.1109/TPAMI.1987.4767941
Hauptmann, M. (2013). Die Gezielte Prozessführung und Möglichkeiten zur Prozessüberwachung beim Mehrdimensionalen Umformen von Karton durch Ziehen [The Trageted Process Management and Possibilities for Process Monitoring in the Multi-dimensional Forming of Cardboard by Pulling], Dissertation, TU Dresden, Saxony, Germany.
Hauptmann, M., and Majschak, J.-P. (2011). “New quality level of packaging components from paperboard through technology improvement in 3D forming,” Packag. Technol. Sci. 24(7), 419-432. DOI: 10.1002/pts.941
Hauptmann, M., Ehlert, S., and Majschak, J.-P. (2014). “The effect of concave base shape elements on the three dimensional forming process of advanced paperboard structures,” Packag. Technol. Sci. 27(12), 975-986. DOI: 10.1002/pts.2089
Hauptmann, M., Weyhe, J., and Majschak, J.-P. (2016). “Optimisation of deep drawn paperboard structures by adaptation of the blank holder force trajectory,” J. Mater. Process. Tech. 232, 142-152. DOI: 10.1016/j.jmatprotec.2016.02.007
Hauptmann, M. (2017). Neue Einsatzpotentiale Naturfaserbasierter Materialien in der Konsumgüterproduktion durch die Technologische Entwicklung des Ziehverfahrens am Beispiel der Verpackung [New Application Potential of Natural Fiber-based Materials in Consumer Goods Production Through the Technological Development of the Drawing Process Using the Example of Packaging], Habilitation, TU Dresden, Saxony, Germany.
Leminen, V., Mäkelä, P., Tanninen, P., and Varis, J. (2015). “Leakproof heat sealing of paperboard trays – Effect of sealing pressure and crease geometry,” BioResources 10(4), 6906–6916. DOI: 10.15376/biores.10.4.6906-6916
Leminen, V., Ovaska, S.-S., Wallmeier, M., Hauptmann, M., Backfolk, K., and Varis, J. (2016). “Effect of material properties and drawing parameters on the quality of deep-drawn paperboard products,” in: 26th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM 2016), Seoul, South Korea.
Müller, T., Lenske, A., Hauptmann, M., and Majschak, J.-P. (2017). “Method for fast quality evaluation of deep-drawn paperboard packaging components,” Packag. Technol. Sci. 30(11), 703-710. DOI: 10.1002/pts.2315
Müller, T., Meyer, M., Lenske, A., Hauptmann, M., and Majschak, J.-P. (2018). “Optical inline quality assessment of deep-drawn paperboard containers,” J. Mater. Process. Tech. 262, 615-621. DOI: 10.1016/j.jmatprotec.2018.08.003
Suzuki, S., and Abe, K. (1985). “Topological structural analysis of digitalized binary images by border following,” Comput. Vision Graph. 30, 32-46.
Tanninen, P., Matthews, S., Ovaska, S.-S., Varis, J., and Backfolk, K. (2017). “A novel technique for the evaluation of paperboard performance in press-forming,” J. Mater. Process. Tech.240, 284-292. DOI: 10.1016/j.jmatprotec.2016.10.002
Vishtal, A. (2015). Formability of Paper and Its Improvement, Ph.D. Dissertation, Tampere University of Technology, Tampere, Finland.
Wallmeier, M., Hauptmann, M., and Majschak, J.-P. (2015). “New methods for quality analysis of deep-drawn packaging components from paperboard,” Packag. Technol. Sci. 28(2), 91-100. DOI: 10.1002/pts.2091
Article submitted: October 12, 2018; Peer review completed: December 15, 2018; Revised version received and accepted: February 2, 2019; Published: February 7, 2019.
DOI: 10.15376/biores.14.2.2536-2549