Workshop Goals

In order to make the workshop both productive and  meaningful for participants, we’d like to collectively define a set of goals and objectives before the workshop itself.

Below is a draft of goals created by the organizers. These are a starting point. Please add your input to this list. If you have additional goals, add them to the post. For revisions or criticisms to the existing goals, please add a comment to the post.  We will refine the structure and schedule of the workshop based on people’s additions and comments.


Distill guidelines and best practices for evaluating digital craftsmanship research
As a group we’d like to define the evaluation criteria which are pertinent to digital craftsmanship research. By comparison we may also define HCI criteria which is irrelevant to the goals of digital craftsmanship research.
Based on these criteria, we aim to distill concrete research methodologies that are useful for evaluating of Digital Craft systems, tools, materials, and environments. This could include methods established in other fields of HCI research like workshops, case studies with expert practitioners, and in-the-wild studies. However it could also include less traditional forms, such as researcher-practitioner collaboration, or evaluation through the researcher’s own experience as a practitioner.

Develop a reading list of exemplary Digital Craftsmanship research.
We seek to compile foundational literature and prior research which can help guide and inform future work in this space. This may include works from HCI, however it may also consist of literature in other domains like art theory, craft history, and anthropology.

Develop a list of active researchers, practitioners, and institutions in the field, and corresponding venues participating in Digital Craftsmanship research.
Who is engaging in HCI research surrounding Digital Craftsmanship? Outside the HCI community? This list can serve as a reference for peer-review and as a vehicle to engage peers in a larger conversation with existing DC research.

Debate and potentially revise the name and definition Digital Craftsmanship.
We selected the name Digital Craftsmanship to collectively describe research which seeks to integrate the qualities of an expressive medium with digital technology. However throughout the workshop we should evaluate this title and definition in comparison to our discussions and conclusions, and revise it as needed.

Define publication venue and contributors for a publication on the workshop results.
We plan to summarize the results of the workshop in a publication and would like to end the workshop with a clear plan for doing so.

Swing Compass: Blending Familiar and Novel Experiences for User Reflection on Selection Habit

Kenny K. N. Chow
School of Design
The Hong Kong Polytechnic University

Swing Compass, developed from one of our research projects, is a configurable, adaptive, and embodied system allowing users to select from a set of items spatially arranged on its interface like a compass rose (Figure 1). Built upon the metaphor: choices “swing from one side to another”, the compass prompts users to turn it with their bodies for different choices, and probably brings about reflection in them on their common selection habits. Examples of everyday selection include deciding what to buy in the supermarket or selecting an app to launch among many others in the mobile phone. Options are displayed on different directions (i.e., N, E, W, S) on the compass rose. Selecting an option is like making a move along that direction. Each option (e.g., tomato) is annotated with attributes (e.g., high fiber) through which the system can compare differences or similarities (e.g., high fiber vs. high protein) between users’ consecutive choices. By default the system suggests options of similar kinds on the compass, that is, having same values in certain attributes (e.g., high protein), compared with the user’s last move. If the user turns with the compass, options of very different kinds show up.

Swing Compass_image
Figure 1.Swing Compass arranges items at different directions on its compass‐rose interface for user selection. Those shown inside the rose are the selection history.

Theoretical Idea and Interpretive Analysis:
Lively Experience Swing Compass is an exemplar of interactive artifacts demonstrating lively reflective potential in users. Lively interactive artifacts enable meaning construction stimulated at multiple cognitive levels, from the immediate, pre-­‐ reflective to the conscious, reflective. The current experience of using a lively artifact is reminiscent of certain past experience from everyday life, yet with nuances that seem curious and stimulate imagination and reflection. Hence, the experience is lively to users both perceptually and mentally. It lies between similarities and nuances, familiarity and novelty, literal and metaphorical. For interpretive analysis, lively experience can be delineated in two stages of interaction, each of which involves four cognitive processes, resulting in different levels of understanding.

At the initial stage of use, it enables immediate understanding of operation:

1. Perceiving affordances – From the case and the compass-­‐rose interface, the user may see it like a big compass or Chinese fengshui compass. One knows he or she can turn with it. The subtle rotation of the compass rose on the interface in accordance with little change in orientation additionally invites the user to turn. This is comparable to Krippendorff & Butter’s recognition and exploration stages [1], as well as Norman’s perceived affordances [2].

2. Receiving quick feedback – The user sees the icons (options) showing up when turning at a certain angle, and then fading out when turning further. One can also turn back a little to bring the fading icons back. The coupling of turning the compass and seeing icons fading forms a motor-­‐sensory feedback loop. The automatic appraisal of the options revealed and the impulsive desire to turn further (new options seem preferred) or back (previous options seem more preferred) mobilize this loop [3].

3. Triggering immediate blends – The act of turning, the fading-­‐in and -­‐out, look familiar to the user, reminiscent of the mundane experience of turning around to look for something. The graphical icons are analogically mapped to the physical targets in the surroundings. The main clash of the analogy is that one has to look down at the icons on the compass, while just looking ahead to the surrounding physical targets. The immediate blend [4, 5] is a new operation of turning to look for something new or different.

4. Becoming second nature – Just after a few turns, the user feels a sense of control. One becomes accustomed to the turning and unaware of the gap between looking down and looking ahead. This resonates with Heidegger’s ready-­‐to-­‐hand [6], Krippendorff & Butter’s reliance stage [1], and Wensveen et al’s natural coupling [7].

If the user sticks with similar kinds of options for consecutive selections without making a turn, the system reduces the available options in order to prompt the user to turn. After a turn, it reveals more choices of contrasting kinds (e.g., high fiber). Through this intervention, the user has to physically swing to open up other possibilities and becomes aware of their unbalanced or biased choice behaviors.

At this later stage of interaction:

1. Noticing unexpected changes – The user notices that not many options are offered and becomes curious. One then opens up more other options after a turn. This is similar to Heidegger’s present-­‐at-­‐hand [6], Lin & Cheng’s “later wow” [8].

2. Invoking interpretive frames – The user tries to make sense of the option arrangement by invoking an interpretive frame [9] about notification of biased behavior. The compass seems to suggest “turning to other underexplored possibilities”.

3. Elaborating metaphorical blends – The intervention evokes some scenarios like parents controlling the choices available to their kid. The reduction of options is analogically mapped with the availability of choices. The resulting metaphorical blend is that indulgence is not recommended and so intervened [10, 11].

4. Appraising and reflecting – The user may feel annoyed by the limitation, or thankful for the reminder.

Major Empirical Findings
Laboratory experiments were conducted with one participant at a time. Each participant was involved in a set of activities, followed by an in-­‐depth semi-­‐ structured interview. The first activity was a warm-­‐up exercise of the compass. The second activity was about using the compass like regular status updating on social media. In the third activity, the participant used the compass as the app launcher for mobile. Some research findings of the study have been published elsewhere [12]. Table 1 shows a summary of the major interview data with some sample quotes from 17 participants -­‐-­‐ ten females and seven males. Five were between 18 and 25 years of age, 11 were between 25 and 35 and one was over 35.

Table 1. Significant responses from 17 participants

Immediate blends Turning as choice making
Switching choices casually in daily life: “It’s like shopping! If you don’t like the items in a shop, or you haven’t seen those before, you turn away!” (XJ)Changing choices with sacrifice: “It reminds me of attending seminar conference last week. Even though I found a presentation boring, I was hesitant whether to move to another concurrent panel. Just like I would not turn because the compass was heavy.” (HU)

Turning as a mental activity
Changing thoughts in problem‐solving: “Turning around when I got stuck in a lecture, in order to find a solution. Turning the body, turning my mind.” (CB)

Unexpected changes and interpretive frames Compromise
“You have to select something you don’t like in order to get something you like.” (TK)

“It’s like I hit the limit and need to relax.” (WW)

Metaphorical blends  About friends’ suggestions
“It’s like a companion, going with you everything. The suggestions might not necessarily be the right one. It’s no harm to listen. Just selective listening.” (KA)

About striking a balance
“If you work on something so often, you become specialized and that may limit your options, you need to diversify your skills … Pendulum -­‐ stops you from being one sided, but it props you to another side.” (RA)

About guidance
“Traffic light, sometimes you need to wait, sometimes it flashes and you need to walk faster.” (WX)

About regulation
“Universities told we could have choices, but actually not many. We can change our majors after the first year, but actually they are all the same.” (WR)

Extended appraisals  Useful and inspirational
“It could be very insightful I think. For food, for health, it’d be great. For apps, I think it’s good as well, because it kind of tells you that you’ve been staring at Facebook for too long … It keeps you from bad choices.” (SC)

All about self-­control
“People don’t like being controlled. Don’t make advice too obvious. Now it’s just right … I try my best to do self-­‐management. I reward myself after doing healthy activities.” (HU)

Too limited
“I don’t like someone set limit on me.” (WW)


[1] K. Krippendorff and R. Butter, “Semantics: Meanings and Contexts of Artifacts,” in Product experience, H. Schifferstein and P. Hekkert, Eds., ed San Diego, CA: Elsevier Science, 2008, pp. 353-­‐376.

[2] D. A. Norman, The Design of Everyday Things. New York: Basic Books Inc., 2002.

[3] P. Ekman, “An Argument for Basic Emotions,” Cognition and Emotion, vol. 6, pp. 169-­‐200, 1992.

[4] G. Fauconnier, “Conceptual Blending and Analogy,” in The analogical mind : perspectives from cognitive science, D. Gentner, K. J. Holyoak, and B. N. Kokinov, Eds., ed Cambridge, Mass.: MIT Press, 2001, pp. 255 -­‐ 285.

[5] K. K. N. Chow and D. F. Harrell, “Elastic Anchors for Imaginative Conceptual Blends: A Framework for Analyzing Animated Computer Interfaces,” in Language and the Creative Mind M. Borkent, B. Dancygier, and J. Hinnell, Eds., ed: Center for the Study of Language and Information Publications, 2013.

[6] P. Dourish, Where the Action Is: The Foundations of Embedded Interaction. Cambridge, Mass.: MIT Press, 2001.

[7] S. A. G. Wensveen, J. P. Djajadiningrat, and C. J. Overbeeke, “Interaction frogger: a design framework to couple action and function through feedback and feedforward,” in DIS 2004 Proceedings of the 5th Conference on Designing Interactive Systems, Cambridge, Massachusetts, USA, 2004, pp. 177-­‐184.

[8] M. H. Lin and S. H. Cheng, “Examining the “Later Wow” through Operating a Metaphorical Product,” International Journal of Design, vol. 8, pp. 61-­‐78, 2014.

[9] C. J. Fillmore, “Frames and the Semantics of Understanding,” Quaderni di Semantica, vol. VI, pp. 222-­‐254, 1985.

[10] G. Fauconnier and M. Turner, The way we think : conceptual blending and the mind’s hidden complexities. New York: Basic Books, 2002.

[11] P. Sengers, K. Boehner, S. David, and J. J. Kaye, “Reflective Design,” in Proceedings of the Fourth Decennial Conference on Critical Computing, Aarhus, Denmark, 2005, pp. 49-­‐58.

[12]K. K. N. Chow, D. F. Harrell, and K. Y. Wong, “Designing and Analyzing Swing Compass: A Lively Interactive System Provoking Imagination and Affect for Persuasion,” in PERSUASIVE 2015 2015, pp. 107-­‐120.

Textiles as On-Body Interactive Surfaces

Noura Howell
School of Information
University of California, Berkeley

My work explores color-changing fabrics and how they can be used to both display information and sense touch, with a particular interest in sensing and displaying physiological signals in daily life. By using textiles in clothing as alternative representations of these signals, I study experiences and interpretations around these artifacts. Textiles as an information display are very different from screen-based information displays, and my work seeks to leverage these material properties to create new associations and interpretations.

Thermochromic Textiles as Information Representation

Screen Shot 2016-05-04 at 9.38.37 PM
Figure 1: Color-changing fabric. Woven and crochet material explorations for leveraging thermochromic textiles to display information, from prior work [2].
Ebb [2] is a collaboration I took part in with Project Jacquard [15] that explores associations around color-changing fabrics and information display in the context of fashion and personal style. What does it mean to wear computationally responsive clothing and why might one wish to do so? Through material explorations [5,16] with weaving and crochet, we developed thermochromic color-changing fabric swatches (Fig. 1). We adapted traditional techniques such as inlay, double weave (for gingham in our case), and simple stripes along the weft of the fabric, to create different clothing-specific designs. Our crochet motifs, which I created, explored material affordances around heating denser or less dense regions, and the interactions of two colors changing simultaneously.

We engaged fashion designers and everyday wearers in envisioning what role these fabrics might play in their design practice and sense of style. Participants volunteered a very different set of criteria when evaluating this technology than one might typically use in evaluating a screen based technology. In particular, participants expressed appreciation for the slowness and low resolution of this material, likening it more to a canvas than a screen, and envisioned slow, gentle, subtle displays. How might these displays shift our interactions with information?

Social Interpretation around a Shirt-Based Representation of Skin Conductance

Screen Shot 2016-03-12 at 10.29.02 AM

Figure 2: Shirt that senses and displays skin conductance. Small white rectangles gradually appear when the wearer’s skin conductance spikes, an indication of excitement such as stress or happiness, from prior work [10].

This work [10] begins to explore the social meaning of clothing-based biosignal representations. I developed Hint, a t-shirt whose thermochromic screenprint pattern responds to the wearer’s skin conductance (Fig. 2), and studied how pairs of friends, each wearing Hint shirts, conversed and interpreted the display. Participants shared a broad range of interpretations, and emotions such as joy and embarrassment were associated with an increase in skin conductance. Additionally, participants expressed desires for their skin conductance displays to help validate their feelings and show emotional engagement with others. In this context, biosignals display became part of social performance [6]. What if wearers crafted their display to support their intended performance? What role might the sharing of biosignals play in social interaction?

Next Steps: Biosensing Clothing in Daily Life

Currently I am exploring thermochromic embroidery, conductive thread for capacitive touch, shape memory alloy, and subtle actuators; different types of garments such as scarves, jackets, and socks; different biosignals such as heart rate, fidgeting, or touch; and ways that clothing already “senses” and “displays” aspects of our bodies and lives with material traces such as sweat marks, a lingering scent of perfume, or wrinkles. Inspired by slow technology [7], I plan to develop a biosensing and displaying garment, and study wearers’ interpretations and experiences with the garment over a few weeks or more.

In the long term with this kind of work, I hope to call out assumptions or unstated trends in biosensing and propose alternatives. As an example, instead of promoting algorithmically defined wellness and self-improvement, perhaps designs could leverage comfortable clothing displays to prompt open-ended reflection or promote self-acceptance of one’s current mental, emotional, and physical state.


1. Laura Devendorf, Joanne Lo, Noura Howell, et al. 2016. “I don’t want to wear a screen”: Probing perceptions of and possibilities for dynamic displays on clothing. Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems (CHI’16).
2. Ylva Fernaeus and Petra Sundström. 2012. The Material Move How Materials Matter in Interaction Design Research. Proceedings of the Designing Interactive Systems Conference, ACM, 486–495.
3. Erving Goffman. The presentation of self in everyday life. Anchor Books, New York.
4. Lars Hallnäs and Johan Redström. 2001. Slow Technology – Designing for Reflection. Personal Ubiquitous Comput. 5, 3: 201–212.
5. Noura Howell, Laura Devendorf, Rundong (Kevin) Tian, et al. 2016. Biosignals as social cues: Ambiguity and emotional interpretation in social displays of skin conductance. Designing Interactive Systems (DIS).
6. Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, M. Emre Karagozler, Carsten Schwesig, and Karen Robinson. 2016. Project Jacquard: Manufacturing Digital Textiles at Scale. Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems.
7. Anna Vallgårda and Johan Redström. 2007. Computational Composites. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 513–522.

Incorporating Kinesthetic Creativity and Gestural Play into Immersive Modeling

Sung-A Jang / / Korea Culture Technology Institute

“I think of immersive virtual space as a spatio-temporal arena, wherein mental models or abstract constructs of the world can be given virtual embodiment in three dimensions and then kinesthetically, synaesthetically explored through full-body immersion and interaction. No other space allows this, no other medium of human expression.” —Chars Davies (1998) [1]

“Computer-generated virtual reality allows the creation of aesthetics that are no longer bound to physical laws and which will become, with faster computers, more real, gripping, and involving. The potential option of perception will be weightlessness, that is, floating through image worlds, touching and transforming computer-generated matter, changes of surfaces and textures, spaces of gigantic and awesome proportions, experienceable individually or collectively, vertig-inous heights and depths, speeds that produce euphoria or paralysis.” —Oliver Grau (2004) [2]

fullviewFigure 1. Left: Spring-mass brush in velocity mode. Variations in stroke weight reflect velocity changes. Center: Experiments in shape-making with the spring-mass brush. Right: Variations in curvatures achieved by adjusting the spring’s dynamic settings.

Myron Krueger, a computer artist and pioneer in virtual reality (VR) interaction, argued that the real power of VR is not in its simulational and illusional capacities, but in its potential to extend our physical reach [3]. In the same spirit, the immersive modeling approach presented in this position paper seeks to use immersive technologies to extend our bodily powers to create sculptural forms in ways that are not possible in the real world. The dynamic models that virtually augment gestural movement in this work are physically inspired and under the creative control we exert over and through our own bodies. The interaction design incorporates sophisticated movements on an intimate scale—the fine-tuned physical control we exert using our hands and fingers. The appearance and behavior of the models are evocative of materials and objects that we are instantly and deeply familiar with and capable of manipulating with dexterity. The ultimate goal is to effectively wield its powers to create visually expressive forms in ways we would not be able to otherwise.

The prototype system presents a new dynamics-driven approach for immersive modeling that enriches the creative process through “gestural augmentation” inspired by physical simulations [4]. It features three new types of dynamics-based interaction techniques for 3D virtual content creation on an immersive platform, which invite playful experimentation and support new forms of expression. This work is part of a broader attempt to bring the richness and spontaneity of human kinesthetic interaction in the physical world to the process of shaping digital form. Immersive technologies and freehand 3D interaction are used to heighten the sense of creative empowerment by blurring the boundaries between the virtual and physical.



leap1  leap2
Figure 2. Left: User’s effective range of activity.
Right: Rift view of a virtual hand model superimposed on infrared imagery.

The system features three different types of hand-based 3D drawing tools which evolved from physical simulations. The parts of the hand that are tracked and used for creating virtual 3D content differ from tool to tool. The 3D modeler uses the tracking information unique to each tool to create virtual 3D forms through key and gestural input. The Rift’s external camera speedily updates any changes in the user’s perceived head position and orientation to seamlessly render the virtual scene within the context of the real world.

The system’s hardware consists of a HMD (Oculus Rift Development Kit v.2 (DK2)) for immersive VR and AR; a motion sensor (Leap Motion Controller) for precise handand fingertip tracking, infrared imagery and gesture recognition; and a high-performance computer that could handle heavy-duty rendering (quad-core 2.6 GHz Intel Core i7 Mac mini). The system’s software was developed with Unity® 3D v.5. Virtual hands, gestural augmentations in action, interface components, sculpted forms, and captured camera images are all combined and rendered as a composite scene for both eyes on the Rift.

Interaction Design

Three types of tools inspired by dynamic models are presented in this system: 1. Spring-Mass Brush, 2. Ribbon Brush, and 3. Rope Tool. The spring-mass brush builds upon Snibbe’s Dynasculpt model [5] and adapts its concept to an immersive platform with freehand 3D interaction. The ribbon brush and rope tool also expand on this dynamic simulation-driven approach and are inspired by real-world physical phenomena.

Spring-Mass Brush


Figure 3. Dynamic model for spring & mass tool.


Figure 4. How drawing progresses with the spring tool.


Figure 5. Cursive writing and a drawing of a bike which show greater degree of control,
made by an artist on his first try after less than an hour of experimentation.


Ribbon Brush


Figure 6. Dynamic model for the ribbon tool.


Figure 7. A ribbon is captured in its current position and another one
immediately appears in its place.

Rope Tool


Figure 8. Dynamic model for the rope tool.



Figure 9. Top: The rope is being held by its two ends and stretched.
Bottom: The rope, once swung up then released, stays frozen in its spot.

3D Palette Interface

All configurable system settings are accessed through the system’s three palettes. In palette mode, three big palettes are placed at an approachable distance and spatially arranged to take advantage of the Rift’s wide FOV. Palettes are arranged in the style of a plane’s cockpit and scaled up to increase accessibility. Users may transition in between drawing and palette mode by pressing the space key. The two modes keep the virtual creations and palettes separate and prevent any overlap or occlusion.


Figure 10. Main menu cockpit layout.

Figure 10 shows the settings of the spring-mass brush, which is the system’s default brush type. The palette on the left controls the 3D brush settings, which allow users to select and adjust the brush’s shape, orientation, scale, extrusion style, and the level of thickness variation. The central palette controls the dynamics settings, which fine tunes the dynamic behavior of the brush. This palette is the cockpit’s centerpiece and its pivotal component. The controls directly affect the critical parameters that govern the dynamic model on which the tool is based. The palette on the right allows users to select the brush’s color and material.


The immersive setting makes it possible for users to create, edit, and experience content directly in 3D. The spring-mass prototype effectively employs a simple dynamic model to virtually augment gestural input and create fluid, expressive strokes. With careful maneuvering, users are able to steer the virtual mass to create curvaceous forms that stretch much further than the finger’s scope of movement. The creation of complex curves, which require extensive training with traditional 3D modeling software, is remarkably easy with this brush. Users generally showed instant familiarity and intuitive understanding of its mechanics, and gained greater command of the brush for more nuanced expressions with practice. The various palettes encouraged and facilitate creative exploration. A conscious effort has to be made, however, to avoid self-occlusion and keep fingertips visible.

Augmenting gestural strokes with dynamic simulations for immersive modeling has proven to encourage creative exploration and experimentation. Gestural interaction and dynamic models drawn from real-world behavior engage our natural spatial instincts and intelligence toward the shaping of form. The brush prototypes successfully combined physically inspired interaction techniques and the unique plasticity the virtual world to extend our expressive capacity and create new forms of expression. Users generally fond the experience was creatively satisfying and liberating.

Each prototype in its current state still imposes a strong visual style specific to its dynamic model. Much more experimentation with dynamic parameters and applicable forces would be necessary to find robust ways of diversifying its creative output. The prototypes would also benefit from taking more advantage of the unique affordances of an AR environment to enable richer forms of interaction that further integrate the physical and virtual.


  1. Davies, Char. “Osmose: Notes on being in immersive virtual space.” (1998): 65-74.
  2. Grau, Oliver. Virtual Art: from illusion to immersion. MIT press, 2004.
  3. Turner, Jeremy, “Myron Krueger Live,”, 25.1-2, Article 104 (2002).
  4. Levin, Golan. “Painterly interfaces for audiovisual performance.” M.S. thesis, Massachusetts Institute of Technology, 2000.
  5. Snibbe, Scott, Sean Anderson, and Bill Verplank. “Springs and constraints for 3d drawing.” In Proceedings of the Third Phantom Users Group Workshop, Dedham, MA. 1998.



The Machine is Learning Aesthetics

Benjamin De Kosnik
California College of the Arts,‭ ‬San Francisco,‭ ‬USA


My long-term project is to watch computers watching television,‭ ‬film,‭ ‬and social media.‭ ‬I call the larger project‭ ‬The Machine is Learning,‭ ‬and it is best thought of as an ongoing set of experiments that seeks to combine computational methods like machine learning and computer vision with media to make art.‭

At the workshop, I will present four of these experiments in closer detail,‭ ‬as case studies.‭ ‬For each study,‭ ‬I will examine the artifacts produced,‭ ‬and discuss the use of software tools in my art practice by answering four questions:‭

1.‭ ‬The choice of hardware,‭ ‬software,‭ ‬and data as defining an aesthetic of computing.‭ ‬Edward Shanken defines aesthetic computing as a hybrid practice that includes‭ “‬a spectrum of inter-‭ ‬and trans-disciplinary research‭” ‬that generates a more symmetrical relationship between art,‭ ‬science,‭ ‬and design‭ [‬1‭]‬.‭ ‬How do I characterize the system or systems that produces these artifacts‭ [‬2‭]? ‬What do these systems have in common,‭ ‬and how do systems from one project differ from the previous project‭? ‬Are the various inputs to the hybrid work in a symmetrical relationship‭?

2.‭ ‬Visual art practice,‭ ‬using art as a tool to do research,‭ ‬and the use of art questions to explore conceptual spaces.‭ ‬What art questions were asked in the creation of the work,‭ ‬and what unresolved questions are present in the final artifacts‭?

3.‭ ‬Data practice and activating the archive.‭ ‬Beginning with Manovich‭ [‬3‭] ‬and continuing on through Whitelaw’s discussion of data practice as an artistic‭ “‬squint‭” ‬through a data space in order to generate a new perspective‭ [‬4‭]‬,‭ ‬the role of artist in perceiving data abstraction is open for interrogation.‭ ‬What data‭? ‬Why‭? ‬How are data limitations,‭ ‬fragments,‭ ‬and subsets used throughout the art making process‭?

4.‭ ‬Software practice and the aesthetics,‭ ‬form,‭ ‬and quality of source code.‭ ‬What methods were used,‭ ‬and how did the experimental procedures evolve‭? ‬How is computation scripted and made into seeing machines‭ [‬5,6‭]?

Case Studies


The Machine is Learning the Man Trap‭ ‬is a two-channel video art installation that uses facial and object detection technology to create new characters and situations in classic media texts.‭ ‬Each screen shows a loop,‭ ‬which is changed weekly.‭ ‬As the weeks pass,‭ “‬the machine‭” ‬leans more about detecting and recognizing faces:‭ ‬simultaneously recognizing more characters with greater accuracy,‭ ‬and yet at the same time negating the human form with algorithmic inscriptions covering the face.‭ ‬By the last week,‭ ‬all faces are completely obscured,‭ ‬covered with an accumulation of algorithmic augmentations.


Valentine Homography uses color contour detection algorithms to look for identical objects in two or more images,‭ ‬applying this algorithm as a Trevor Paglen-defined seeing machine to social media images:‭ ‬profile portraits,‭ ‬liked images,‭ ‬and disliked images.‭ ‬This computational lens simulates computer matchmaking in a visual form that represents one instance in a sea of many thousand‭ “‬known-good‭” ‬or positive test cases that the machine learning behind websites such as Facebook,‭ ‬Tindr,‭ ‬Grindr,‭ ‬OkCupid,‭ ‬,‭ ‬,‭ ‬etc.‭ ‬must crunch to create a single user’s match.


Equal Weight Uhuras‭ ‬is a six-channel video art installation that uses facial recognition algorithms to quantize six characters from the‭ ‬1966-1969‭ ‬television series‭ ‬Star Trek,‭ ‬and reconstructs all three seasons in a condensed form.‭ ‬In the reconstructed form,‭ ‬the characters Spock,‭ ‬Kirk,‭ ‬Uhura,‭ ‬Scotty,‭ ‬McCoy,‭ ‬and Sulu all have the same amount of‭ “‬screen time‭” ‬as the character Uhura,‭ ‬with the added proviso that all the characters are shown talking equally to each other in a random fashion,‭ ‬or conversing with a representation of space.‭ ‬This perspective illustrates the differences in the quantity and quality of media representations by gender and skin color.


In‭ ‬Marihuana Smokes Herself,‭ ‬clips extracted from film and television depict cannabis use,‭ ‬production,‭ ‬and distribution.‭ ‬My practice is to watch media and note any cannabis on the screen in a database,‭ ‬as if I were a media censor in a time and space of total cannabis prohibition.‭ ‬Over time,‭ ‬this process has created a growing archive of clips that depict cannabis as a character in the larger war on drugs.‭ ‬ÓThe sequenced clips in this three-channel video art installation use this archive,‭ ‬but only the parts that contain a female character on the screen.‭ ‬This subset is further filtered by interaction with viewers and creating new loops from suggested searches such as‭ “‬numb,‭” ‬power and money,‭ ‬dealing,‭ ‬actual medical use,‭ ‬and others.


1.‭ ‬Shanken,‭ ‬Edward A.‭ ‬Draft of Aesthetic Computing,‭ ‬Forthcoming in Oxford Encyclopedia of Aesthetics,‭ ‬2015

2.‭ ‬Jones,‭ ‬Caroline A.‭ “‬System Symptoms:‭ ‬Caroline A.‭ ‬Jones on Jack Burnham’s‭ “‬Systems Esthetics‭” (‬1968‭)‬.‭” ‬Artforum International‭ ‬1‭ ‬Sept.‭ ‬2012:‭ ‬113-14,116,534.‭ ‬Print.

3.‭ ‬Manovich,‭ ‬L.‭ “‬The Anti-Sublime Ideal in Data Art Visualization and Mapping.‭” ‬Web.‭ ‬4‭ ‬May‭ ‬2016.‭

4.‭ ‬Whitelaw,‭ ‬Mitchell.‭ “‬Art Against Information:‭ ‬Case Studies in Data Practice.‭” ‬The Fibreculture Journal‭ ‬11‭ (‬2008‭)‬.‭ ‬Web.‭ ‬4‭ ‬May‭ ‬2016.‭

5.‭ ‬Paglen,‭ ‬Trevor.‭ ‬III Scripts,‭ ‬,‭ ‬last accessed‭ ‬2015-04-05

6.‭ ‬Paglen,‭ ‬Trevor.‭ ‬II Seeing Machines,‭ ‬,‭ ‬last accessed‭ ‬2015-04-05

Making Musical Mates: A story of digital craftsmanship


Andrew R. Brown
Griffith University, Brisbane, Australia

In Australian colloquial language, a ‘mate’ is a friend or companion. More broadly a ‘mate’ can indicate a more intimate, even sexual, relationship. The term, therefore, works well to evoke the kind of close partnership I have in mind between a musician and their musical instrument. But what does it mean to design and construct such an instrument, to make an interactive music system with a sense of creative agency that evokes a rich sense of creative partnership? What, in short, is the craftsmanship required to make a digital musical mate? This article discusses some of the issues that arise in answering these questions from both a conceptual perspective, and as creative practice activities in designing, making and playing original musical devices.

Post-Digital Musicianship

Digital musicianship is, according to Andrew Hugill is a different kind of artistry from traditional musicianship, it is “a disembodied form of knowledge made evident through a set of technical skills and judgements” (2012: 52). Even while knowledge may be ‘disembodied’, the skills of the digital musical often include an embodied from of making; from connecting equipment together to allow for appropriate signal flow, to programming patches and scripts to customise software systems.

The making of music in the electronic age requires controlling and managing musical machines and experimenting or hacking devices in what are sometimes called “post-digital” practices (Cascone 2000). Like all musicians, the accomplished digital musician exercises expertise in a manner consistent with Richard Sennett’s descriptions of craftsmanship. “Every good craftsman conducts a dialogue between concrete practices and thinking; this dialog evolves into sustaining habits, and these habits establish a rhythm between problem solving and problem finding” (2008: 9).

The experimental, or “post-digital” margins of this kind of musicianship are of particular interest in this article. When Kim Cascone coined the term post-digital, he characterised associated practices as arising out of the ‘failure’ of technologies, an exploitation of their glitches and faults to artistic ends. While embracing breakdown might seem to undermine any sense of ‘control’ that is associated with mastery, my sense is that the true nature of materials (digital included) is revealed in their breakdown. That an awareness of the boundaries of a medium and pushing beyond routine uses is a part of any quality craftsmanship. As Malcolm McCulloch observed, “[o]ur use of computers ought not be so much for automating tasks as for abstracting craft” (1996: 81). In this way digital tools, namely computers, become an expressive media in the hands of an accomplished digital musician; including, in this definition of musician, an instrument maker.

Why build things?

According to Philip Agre, “what truly founds computational work is the practitioner’s evolving sense of what can be built and what cannot” (1997:11). The concrete experience of assembling electronic components or wrangling software syntax and algorithmic descriptions, grounds the maker in pragmatics of the craft. The kind of practical and embodied experience of such making constitutes an important way of knowing.

Though doing the work of constructing new musician instruments, the processes of making become intertwined with the structures of music. Decisions about either, impact on decisions about the other. The roles of designer and user are coupled, and any distinction between them is especially pointless when a single person takes on both roles; as is often the case in experimental cultures and post-digital practices.

The influences on making extend beyond the designer and the user. Importantly, the technocultural context of making is also critical. The environment in which we make often dictates resources, imposes cultural conventions, and is a source of interpersonal interactions that impact upon the meaningfulness of the activity. Often what can be build, is not only the device but the environment that enables the making itself.

Case Study examples

In this next section I will describe two examples of musical mates from my own work. Both are interactive music systems; semi-autonomous musical machines designed to act as musical partners with a degree of agency that goes beyond a traditional definition of ‘musical instrument’. These descriptions are only brief but they put flesh on the bones of the surrounding discussion and provide the reader with a grounding in the particular. In essence the examples provide the article with a similar pragmatic engagement with materiality that the technologies do for the digital instrument maker.


CIM, pronounced Sim, is a virtual improvising musical partner. It consists primarily of software that ‘listens’ to what a musician plays and performs a response based on that input. Such an architecture classifies CIM as a ‘reflexive’ music system (Pachet 2006). The software includes analytical and generative functions based, in part, on theories of music, music perception and improvisation interactions. Typically CIM is used with the performer playing a MIDI controller (say an electronic keyboard) and sound can be generated on a synthesizer. In much of the development work with CIM it has been connected to two MIDI-controlled acoustic pianos that provide a degree of musical ‘authenticity’ and interest to performances. Additionally, CIM includes some interface elements for the performer; typically a set of foot pedals that provide limited control over CIMs behaviour, and a mobile tablet that displays visual cues about state of the software. Mostly visual cues reinforce what the musician can hear in CIM’s performance, but at times visual feedback allows the musician to predict some of CIMs action. At other times the display provides a substitute for the visual cues that would exist between human performers. Various examples of performances using CIM are available online;

The 8 Box

This device is a small handmade electronic musical instrument. It is developed using resources of the ‘maker’ movement and in an attempt to engage more directly with the materiality of music technologies. It’s also a vehicle to explore how to program a simple computing device with varying degrees of musical agency. As a result, interactions with the 8 Box can range from direct gestural performance, through indirect navigational control of sound worlds, to allowing autonomous generated sonic output.

Based on a Teensy board—an Arduino-like micro controller—the 8 box employs sound output direct from the PWM pins of the board and explores the sonic and musical potential of only having two parameter controls; pulse width and frequency. This sound source results in a raw and edgy timbre, peppered with occasional analogue and digital glitches and various emergent properties. This is characteristic of ‘post-digital’ practices articulated by Cascone (2000) and expanded upon in Ferguson and Brown (2016). As with all Arduino projects the software for the 8 Box is written in C. Coded processes manage pin oscillation, sound parameter control, musical structure and timing, and user interaction. More details about and media of the 8 Box are available online at

Agency Network

An underlying principle in the design of musical mates, such as CIM and The 8 Box, is taking into account all of the contributors to the final musical outcome. These contributors exist well beyond the obvious elements of musician and device. Other contributors include cultural conventions of music practice, musical genres and stylistic rules, theories of music and aural perception, and likely physical performance space and technical requirements for presentation. Each contributor may elicit agency because they ‘cause things to happen’ to varying degrees. Such causality is, according to anthropologist Alfred Gell (1998), the basis for attributing agency. Despite a lack of conscious intentionality, devices and other objects can exert a degree of agency by way of their influence within an interactive system that includes an intentional stance provided by human participants. Such a multi-faceted system of interacting human and non-human contributors I call an agency network (Brown 2016); with a nod to Latour’s actor network theory (2005).

Agency networks acknowledge that craftsmen take account of the affordances of their surroundings, their materials, and the contexts in which their creations will exist. It takes account of the situatedness of activities. It resonates with ideas of a distributed self, where consciousness and intelligence leverage other people and the features of surrounding materiality. In many cases it can be hard to decipher where the inspirations arise, where influences come from, or what agencies are active within the network. Indeed it is often the case that “the line between human intention and material affordance becomes all the more difficult to draw” (Malafouris 2008: 33). Just as this was true for traditional craftsmen who surrounded themselves with a workshop environment, specialised tools, quality materials, and a guild of fellow artisans; so it is in the digital age. As McCullough writes in his book about digital craftsmanship, “In all likelihood, a human-computer partnership will continue to surpass either unaided human or the autonomous algorithm for some (important) aspects of work” (1996: 103).


The skill of designing and building new interfaces for musical expression involves a digital musicianship, a craftsmanship, that includes skills and judgements different from those in traditional musicianship. The making of musical instruments has long been a venerated practice and the craftsmanship of making in the digital (or post-digital) age continues this tradition. The work still requires an understanding of the music discipline and an appreciation for the materiality of digital and electronic media. The practice invites, and at time requires, a motivation born of enquiry and a skill approaching virtuosity. Sennett describes such acts as “engaged material consciousness” through which “we become particularly interested in the things we can change” (2008:120).

I have suggested here that this extended consciousness can extend further than to the material agency of physical media and on to the social, cultural and theoretical contexts in which we make. These aspects combine into an agency network with a complex web of relationships that we navigate and manage.

Ultimately, however, the making of a ‘mate’ is the making of an ‘other’. It is quite likely that the other is formed in out likeness. Either directly, as in the reflexive processes of the CIM software, or less overtly, like the cultural and ergonomic choices of the 8 Box product design. Taken as such, “a mirror-tool—my coinage—is an implement that invites us to think about ourselves” (Sennett 2008: 84). In an interesting turn of perspective we can appreciate that as craftsmen we are not simply reflecting our values in our work but, in the spirit of constructivism, we acknowledge that in making our musical mates we are, in a way, making ourselves.


Brown, A. R. (In press). Understanding Musical Practices as Agency Networks. In Proceedings of the International Conference on Computational Creativity. Paris.

Cascone, K. (2000). The Aesthetics of Failure: “Post-digital” tendencies in contemporary computer music. Computer Music Journal, 24(4), 12–18.

Ferguson, J., & Brown, A. R. (In press). Fostering a post-digital avant-garde: Research-led teaching of music technology. Organised Sound.

Gell, A. (1998). Art and Agency: An anthropological theory. Oxford: Clarendon Press.

Hugill, A. (2012). Musicianship in the Digital Age. In A. R. Brown (Ed.), Sound Musicianship: Understanding the Crafts of Music (pp. 52–61). Newcastle upon Tyne: Cambridge Scholars Publishing.

Latour, B., 2005. Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford: Oxford UP

Malafouris, L. (2008). At the potter’s wheel: An argument for material agency. In C. Knappett & L. Malafouris (Eds.), Material Agency: Towards a Non-Anthropocentric Approach (pp. 19–36). Springer.

McCullough, M. (1996). Abstracting Craft: The practiced digital hand. Cambridge, MA: The MIT Press.

Pachet, F. (2006). Enhancing individual creativity with interactive musical reflexive systems. In I. Deliège & G. A. Wiggins (Eds.), Musical Creativity: Multidisciplinary Research in Theory and Practice (p. 359). New York: Psychology Press.

Sennett, R. (2008). The Craftsman. London: Penguin Books.

Older Adults with Cognitive Impairments as Capable, Competent, and Engaged Artists

Amanda Lazar1, Caroline Edasis2, Anne Marie Piper1

1 Northwestern University, Evanston, IL 2 Mather Lifeways, Evanston, IL,,


Older adults with cognitive impairments such as dementia are often stigmatized and marginalized by society. This is in part due to a biomedical understanding of cognitive impairment [1], which sees people with dementia as having a “brain [that] has been destroyed by the disease and who therefore no longer exists as a person but only as a body to be managed” [2]. This view leads to older adults with dementia being regularly prevented from using their abilities to the full extent possible, having their actions and expressions interpreted as a negative side effect of dementia, and receiving damaging messages to self-esteem [3]. When we view dementia through the lens of the social model of disability, we can see that the socio-cultural context we live in is responsible for the physical and social barriers that hamper the ability of people with dementia to engage fully in society – in other words, people have designed the world we live in for those without cognitive impairments [4]. However, some environments are designed for people with cognitive impairments and enable individuals to use their abilities to the fullest extent: one such setting is art therapy. We have conducted over two years of fieldwork on the practice of making among older adults with cognitive impairments within art therapy. We have observed group and individual art therapy sessions at one retirement community and conducted thirteen interviews with art therapists who work with older adults in a variety of settings. Our analysis reveals the ways in which the practice of art therapy positions older adults with cognitive impairments as capable, competent, and engaged through the act of creating artwork. Below, we describe how creating in art therapy empowers older adults with cognitive impairments, is a collaborative process, and focuses on social interaction rather than the final artifact.


Empowerment through making

Art therapy provides a carefully constructed setting that empowers older adults with cognitive impairments. To understand how this setting is created, we draw on the metaphor of the Third Hand, coined by the renowned art therapist Edith Kramer. An effective art therapist acts as a third hand to the client’s two hands, and “helps the creative process along without being intrusive, without distorting meaning or imposing pictorial ideas or preferences alien to the client” [5]. Through our fieldwork, we found four ways that an effective Third Hand interacts with clients with cognitive impairments in art therapy: providing dynamic support; fading into the background to remain invisible and fluid; promoting autonomy and choice; and confirming abilities and self-worth [6]. The art therapist, acting as the Third Hand, works with clients with a wide range of abilities and dynamically adjusts to fluctuations in cognitive and physical abilities. In doing so, the emphasis is not on filling a void or deficit resulting from impairment. Instead, the Third Hand views the environment as problematic rather than the individual [4] and addresses this environmental mismatch by modifying the environment to support each individual. This is done in a variety of ways: the therapist provides functional assistance by loading a brush with paint or steadying a client’s hand; the therapist recommends art materials that best match the client’s abilities in that moment, such as water colors to minimize resistance between the brush and paper for a client whose shoulder was hurting her. Another way the therapist constructs a supportive environment is through confirming ability and self-worth of individuals with cognitive impairment who, outside of art therapy, often are treated in ways that minimize feelings of self-esteem [3]. One way the art therapist does so is by using language and materials that indicate that making art is a serious activity. The therapist refers to the clients as artists, in contrast to visiting family members and friends who drop by and refer to the artwork as ‘crafts’. The therapist regularly clarifies that this is an “art group” and that “it’s a good group of artists we have here.” Calling attention to this work as “art” emphasizes the deep reflection that is part of the therapeutic process of making artwork and the encouragement artists are given to express themselves through their work. With the client’s permission, the art therapist frames their artwork in high quality frames and exhibits their work in the community gallery spaces, which further emphasizes that what these participants are creating is valuable and worthy of sharing.


Collaborative nature of making

In contrast to a view of art as an individualistic activity, the art created during art therapy is inherently collaborative. Acting as the Third Hand, the art therapist serves as an involved facilitator, setting up a customized workspace for each client and finding specific materials that will suit the interests of different clients. For example, she purchased a chalk staff liner for a client who had been a musician, and the client incorporated lines drawn with it into her art piece. The therapist may also provide prompts for art making, set finished artwork in front of the client for inspiration for the next piece, and bring up themes that a client discussed in sessions beforehand. The art therapist will provide contextual information necessary to help clients recognize artwork and re-enter the making experience when a client does not recognize the artwork or remember details about it from week to week [6]. The art therapist also is a co-creator in the making experience – during an interview, an art therapist noted that they will sometimes work with a client to make art, asking “‘Can I collaborate with you to help you finish this?” When art is included in an exhibit or digitized, the art therapist may decide to only use a part of the canvas or digitally brighten or increase contrast of the picture. The therapist is also a co-owner of the art as they are the ones who collect, store, and make many of the decisions around sharing the artwork [7]. In contrast to an individualistic view of making, art making in art therapy is facilitated by the art therapist, and art that is created in the session is co-created and co-owned by the client and therapist.


Social interaction through making

Art therapy allows for different types of interactions that enable older adults with cognitive impairments to participate socially in ways other environments do not support. The art therapist at our fieldsite explained:

“If [the client] can have those opportunities to connect with people and be treated like a person that is a huge thing… Art plays a really big part in addressing that, because when you’re having a creative interaction with somebody, you’re not bound by the normal linear conversation, social[ly] acceptable rules of how people normally interact. Because you are expressing yourself through multiple forms, you’re engaging with somebody in a more playful spontaneous way, you don’t even necessarily have to be able to speak coherently to be able to express yourself creatively.”

During making in art therapy, social rules that prize logic and reasoning become less important. Sensory experiences, humor, and emotional awareness, all continuing strengths that people with cognitive impairment retain long into the condition [8], are valued. The strengths of people with cognitive impairments are employed in art making, and “symptoms” of dementia are viewed as expressions of the self to be engaged with creatively. People with cognitive impairments are empowered to make decisions, express opinions, and create in an environment where what they say is not challenged or disregarded. Thus the process, or what is expressed throughout the process, is particularly important, arguably even more so than the final artifact. The art therapist at our fieldsite brought up an example of a client she worked with:

One of her pictures is this green amorphous shape, but if you had the audio to go along with it, you would understand that she had been talking about the green dress that she wore when she got married. And so suddenly that picture is communicating a lot more than you initially would think.

Making art is a way in which older adults with dementia express themselves, particularly when they are unable to speak, and their artwork can provide an entry point into social interaction with their peers, family members, and community staff.


What is the role of technology?
It is worth noting that none of the older adults we study in art therapy use computers. Hence, in studying this relatively technology-free practice of making, we question what role digital technologies could play in enhancing this work. Incorporating technology could enable these artists to share their artwork with a broader audience and provide the ability to incorporate new modes of expression, such as audio or video, that could also help capture the process of artmaking. However, bringing technology into this setting can also disrupt this carefully constructed environment. How might technologies support making in this context without distracting the therapist from carefully attending to her artists’ needs? How do we retain the benefits of the physicality of the art making experience, which draw on the strengths of these artists? Most importantly, how do we keep the art activity accessible and empowering, particularly when computer interfaces may highlight individual limitations?


  1. Nancy Harding and Colin Palfrey. 1997. The Social Construction of Dementia: Confused Professionals? Jessica Kingsley Publishers.
  2. Susan M. Behuniak. 2011. The living dead? The construction of people with Alzheimer’s disease as zombies. Ageing and Society 31, 1: 70-92.
  3. Tom Kitwood. 1997. Dementia Reconsidered: the Person Comes First. Open University Press.
  4. Murna Downs. 2000. Dementia in a socio-cultural context: an idea whose time has come. Ageing and Society 20, 3: 369-375.
  5. Edith Kramer. 2000. The Art Therapist’s Third Hand: Reflections on Art, Art Therapy and Society at Large. In Art as Therapy: Collected Papers. Jessica Kingsley Publishers, 47-70.
  6. Amanda Lazar, Raymundo Cornejo, Caroline Edasis and Anne Marie Piper. 2016. Designing for the Third Hand: Empowering Older Adults with Cognitive Impairments through Creating and Sharing. To appear in Proceedings of the ACM Conference on Designing Interactive Systems (DIS ‘16).
  7. Raymundo Cornejo, Robin Brewer, Caroline Edasis, Anne Marie Piper. 2016. Vulnerability, Sharing, and Privacy: Analyzing Art Therapy among Older Adults with Dementia. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW ‘16).
  8. Carol Bowlby Sifton. 2000. Maximizing the Functional Abilities of Persons with Alzheimer’s Disease and Related Dementias. In Interventions in Dementia Care: Toward Improving Quality of Life. Springer Publishing Company, 11-36.


Living on the edge: supporting continuous knowledge seeking among professional digital artists

Volodymyr Dziubak
Ph.D. student
Department of Computer Science,
University of Manitoba,

Dr. Andrea Bunt
Associate Professor,
Department of Computer Science,
University of Manitoba,

It would be difficult to imagine modern digital craftsmanship without artists continuously learning and mastering the software tools they use to create digital artifacts. From year to year, software companies introduce numerous changes into their digital products. In the frequent updates and patches, they release new features as well as re-envisions of some of the existing functionality. In addition to changes in the software itself, creative individuals invent new ways of using the functionality of their digital tools to achieve a variety of artistic effects. Such creative and often unexpected techniques generate a lot of interest among expert software users, who are constantly looking for ways to make their workflow more diverse and efficient [1]. Considering the two scenarios we outlined above, namely, the constantly changing software and the appearance of new techniques within its community of users, it is no wonder professional digital artists actively seek out new things they could learn about the software they use. Our research is aimed at understanding and supporting such continuous learning among professional users of digital software, particularly for those using the tools for creative work, such as artists and designers.

Whereas prior work on software learning has mainly focused on learning activities motivated by a specific task (e.g., [2, 3]), our research focuses specifically on continuous, task-free software learning [4, 5], motivated by curiosity and desire to stay up-to-date with recent news, techniques, and tricks. Specifically, we study current learning practices among professional digital artists and designers, find limitations of those approaches, develop tools that address those limitations, and evaluate their effectiveness based on practitioners’ satisfaction.

In our recent work [1], we investigated how displaying rich information clues alongside tutorials posted on social media can facilitate serendipitous discovery of previously unknown techniques. Our initial interviews with expert designers and artists revealed that discovering new information and inspirational techniques most often happens by accident. Without having a specific learning goal in mind, professional designers and artists habitually browse social media news feeds and forums, looking for something that could inspire them, surprise them, or in any other way catch their attention. For example, our interviewees reported watching tutorials they discovered while browsing through their Twitter feeds, in hopes of learning something new in the video, or in the related videos. However, we also found that even though the number of tweets referring to software tutorials is rather high (informally, we found that the number of tweets with tutorials for a specific creative software could reach as high as 30 in one hour), the limited information provided in each tweet makes it difficult to judge if the tutorial is worth watching or not. Having already substantial expertise with the software, digital artists do not want to spend time watching a tutorial unless they are confident it has something they might consider new or inspiring. At the same time, many participants reported that it is very difficult to determine whether a tutorial had inspirational content without watching it first. To address this and other limitations we implemented Switter – an alternative Twitter client that provides rich tutorial-related information cues alongside tweets with tutorials.

Switter’s interface: augmented Twitter news feed with the replica of Photoshop’s interface

Switter approaches the problem of insufficient information about tutorials in tweets by introducing two types of information cues. First, below each tweet, Switter shows the list of Photoshop commands and tools used in the mentioned tutorial. These brief information snippets provide users with a minimalistic overview of the techniques used in the tutorial without the need to watch the tutorials first. Second, Switter provides a novel way to navigate through the large volumes of tweets with tutorials by embedding a replica of the target software’s interface in a Twitter client. We chose Photoshop as our target software, so the interface replica was comprised of all of Photoshop’s commands and features from the main menu, the main toolbar, and floating panels. Each of the commands and features acted as navigation – selecting it would filter the list of tweets with tutorials to only display those that involve the selected command or feature. Next to each feature in the interface replica Switter also shows an indicator of how many tutorials use the feature. As we discovered in our field study, these indicators facilitate discovery of unknown commands and enable focused exploration on specific skills.

Evaluating Switter in a weeklong field study with nine Photoshop experts revealed a variety of scenarios where the information cues provided in Switter helped participants discover and learn new things about Photoshop. First, the minimalistic overview of the commands used in each tutorial allowed participants to compare their own approaches and techniques to the ones used in the tutorial, helping them to dismiss tutorials that did not contain anything they could learn, as well as to notice inspiring content in tutorials they would have dismissed otherwise. Second, the awareness indicators next to each command in the interface replica guided participants to new and inspiring content. For example, by following the indicators with the highest numbers, participants discovered and learnt about features they did not know existed. Finally, selecting specific commands in the interface replica allowed participants to look for techniques that involved specific tools, with the purpose of either improving on their weaker skills, or comparing several tools in terms of their effectiveness and use cases. Overall, the study showed that providing minimalistic information cues might help expert software users find new inspirational software techniques and tricks when they browse through social media.

Our evaluation of Switter was only the first step towards understanding and supporting continuous information seeking among digital artists. Switter helped our participants to identify tutorials that cover inspirational or unexpected techniques, but it did not allow them to follow their discovery and get more information about the source of the inspiration. As an example, the tutorial might be a part of the entire series of tutorials covering a specific topic that the expert is interested in. Alternatively, the technique covered in the tutorial might showcase the use of a new feature introduced in the latest Photoshop update. In both cases, connecting the content to its source and to other content with the same origin could lead digital artists towards further discoveries of new and interesting ways to use their tools. In the future we plan to explore ways of uncovering connections between potentially inspirational content, such as tutorials, news articles, official changelog pages, etc. Once the connection is identified, the next step would be to visualize these relationships in a way that would allow software experts to not only identify new and unexpected information, but also to follow the connections and find more related and potentially inspiring content.

Overall, our research investigates how expert digital artists stay up to date with the most recent news, learn new tricks, and advance their skills with the digital tools they use every day. In doing so, we aim to help digital artists master the tools that they use as an integral part of their artist work.

  1. Volodymyr Dziubak, Patrick Dubois, Andrea Bunt, and Michael Terry. 2016. Switter: Supporting Exploration of Software Learning Materials on Social Media. (To appear in) Proceedings of the 2014 conference on Designing interactive systems – DIS’16
  2. Caitlin Kelleher and Randy Pausch. 2005. Stencils- Based Tutorials: Design and Evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems – CHI’05
  3. Gierad Laput, Eytan Adar, Ann Arbor, Mira Dontcheva, and Wilmot Li. 2012. Tutorial-Based Interfaces for Cloud-enabled Applications. Proceedings of the 25th annual ACM symposium on User interface software and technology – UIST’12
  4. Brian Dorn and Mark Guzdial. 2010. Learning on the Job: Characterizing the Programming Knowledge and Learning Strategies of Web Designers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems – CHI’10
  5. John Rieman. 1996. A Field Study of Exploratory Learning Strategies. ACM Transactions on Computer- Human Interaction – TOCHI’96

H(EC)I – Human Ethno-Computational Interaction: (Re)conceptualizing Wire-Bending and Design in the Trinidad Carnival

Vernelle A. A. Noel

Ph.D. Candidate

Department of Architecture

The Pennsylvania State University

Investigating the ways in which humans engage with digital technology, and the development of novel interfaces for making use of technology are key contributions of HCI. Since it focuses on people’s interactions with computers, it might be assumed that by incorporating digital tools into non-digital environments, insight may be gained into the power and limits of new technologies.  Despite recent investigations into craft, particularly in discourses on digital and hybrid making, the interactions between craft persons, their tools, artifacts, and their communities are generally positioned as outside the boundaries of digital making. Thus, the socio-technical processes and cultural meanings involved in these maker cultures are not usually brought to the forefront.

In this position paper, I argue that a comprehensive understanding of craft-persons, the ways in which they interface with their tools, artifacts, and each other, as well as an understanding of their beliefs, can yield valuable insight to inform and demarcate this potential field of Digital Craftsmanship – specifically by (re)conceptualizing the term “digital” as one with real social, historical, and cultural dimensions, i.e. computational. I call this approach H(EC)I for Human Ethno-Computational Interaction. Using the craft of wire-bending, and design in the Trinidad Carnival as the site for my study, I demonstrate how traditional (non-digital) craft and digital practices can merge through ethnography, computation, and interaction through this approach – from culture to technology.

I grew up in a family where making was, and still is, an important part of our culture – making by hand. I learned through making, and made to learn. I enjoyed nothing more than engaging my hands and my brain in making. (TEDx Talks 2016; Schumacher 1989) In my training as a computational designer, I have learned how to think, make, and design computationally, and digitally. My work in craft, computation, and technology is motivated by my love of making, my interest in interactions embedded in making cultures, empowering people through familiar signs and symbols, and introducing computation and digital technology via culturally-relevant support. The current gap between traditional (non-digital) and digital making is more than a technological one, it is a cultural one. Western versus non-Western, making by hand, versus making by computers.

To reconcile this gap, we need to first understand the ways in which non-digital cultures engage with their tools and materials to create objects, and the problems they may be experiencing. Secondly, we need to develop culturally-relevant interfaces and systems for advancing their craft in their communities. Thirdly, we develop systems that integrate craft interfaces with digital technology. Let’s briefly consider my attempt to reconcile technology and craftsmanship using the practice of design in the Trinidad Carnival, and the craft of wire-bending as an example. The Trinidad Carnival was invented by newly emancipated slaves in 1834 to celebrate their freedom, express their creativity and aesthetic sensibilities (Brown 1990). Cultural practices, meanings and social conditions are articulated through the carnival (Lee 1991; Ryan and Institute of Social and Economic Research 1991); it forms and sustains local communities (Riggio 2004); creates a sense of pride (Miller 2013); brings different generations and people together; and positively impacts local and global economies (Tull 2005). The Trinidad Carnival is culturally, historically, and economically important to Trinbagonians, and the diasporic communities.

The craft of wire-bending is a “specialized art, combining elements of structural engineering, architecture, and sculpture” which developed in the 1930s in carnival in Trinidad and Tobago (Velasquez 2016; Bailey 2013). Wire and other thin, flexible strands of material are bent and assembled to create two-dimensional (2D) and three dimensional (3D) structures. Through this craft, people interface with tools and materials to create objects.

(Human) in Ethnography

Through ethnographic methods I examined sites, conducted interviews with designers and craftsmen, inspected wire-bent artifacts, and observed the ways in which craftsmen engaged with their tools and materials to create these large artifacts (Figs. 1 and 2). By carrying out the above I developed a comprehensive understanding of the community and what making means to them. I gained insight into the meaning and importance of, as well as the problems being experienced in designing and making in carnival. One problem is the lack of involvement by communities in designing and making artifacts for carnival. Consequences of this are the loss of local knowledge, the lack of development of new structural forms, techniques, and knowledge in design. Additionally, the indigenous practice of wire-bending is dying due to the slow rate of transmission of this knowledge, its labor-intensive nature, and the younger generation’s love of technology (Noel 2014; Noel 2013). Here I illustrate the first point of my argument, understanding what making means to this community, examining the ways in which craftsmen interface with their tools and materials, and articulating the problems experienced by that community.

Figure 1. Materials used in wire-bending
Figure 2. Tools and processes in wire-bending

(Human) in Computation

After acquiring a comprehensive understanding of the situation, I then explored opportunities to develop culturally-relevant design support. I developed the Bailey-Derek Grammar (Fig. 3).  – named after two local, expert wire-benders, Albert Bailey and Stephen Derek. This computational system which codifies the main techniques and spatial relation in wire-bending was developed to capture, explicate and reinvent vernacular knowledge in wire-bending. Externalizing this knowledge enables the recording and preserving of culture, its transmission to others, and the development of existing knowledge. This grammar is represented in the form of drawings, a familiar form of communication (signs and symbols) used by the craftsmen. This novel computational tool illustrates the second point of my argument, that of developing culturally-relevant interfaces and systems for advancing the craft of those communities.

Figure 3. An example of the Bailey-Derek Grammar codifying techniques and steps in wire-bending

(Human) in Interaction

Interaction with design ideas allow users to engage in simulated conversations with designs that can inform design decisions, and allow the exploration of “spatial and temporal movement” (Visser 2009; Kalisperis 2002). Including computer-aided tools in the design process can enable idea generation, exploration, and evaluation of design alternatives, thereby reducing total reliance on full scale, physical prototyping for feedback, thereby having a positive impact on design outcome (Blessing 1995). By creating rule-based descriptions of designs and implementing them in a computational environment, I was able to create several design alternatives. Human interaction with these designs through computers creates a new space for those interested in digital technology and design.

In this stage of the work, I tested the interaction between a chosen design and the body to explore how the interaction might impact or inform design and design thinking. I used a visual programming language (Grasshopper and Firefly) along with an input/ output motion-sensing device (Kinect) to allow interaction between the body and the design (Fig. 4). Since these artifacts are performed by the body in carnival, the ability to visualize and simulate its movement with the body was paramount. The development of this interface illustrates my third point. It integrates digital technology to address the issue of the younger generation’s love of technology, and the simulation of design to aid in design exploration. The interface integrates the cultural practice with digital technology (Noel 2016).

Figure 4. User/ Designer interacting with designs with motion-sensing technology
Figure 4. User/ Designer interacting with designs with motion-sensing technology


This example of the H(EC)I approach to reconciling technology and craftsmanship highlights the importance of understanding the values of different cultures; developing computational, culturally-relevant support which may be digital or non-digital; and creating interfaces that merge the computational with the digital.(Vernelle Noel 2016) If we are to understand how to reconcile technology and craftsmanship, and enable diverse forms of expressive practice by many different people, then we must carry out ethnographic examinations to understand the meaning, importance, and problems that exist in these cultures, with the goal of empowering them. (Eglash et al. 2006) After understanding these problems, only then can we develop culturally-relevant support that would also inform HCI. My purpose is to suggest that this H(EC)I approach to digital craftsmanship as a field of research in HCI could benefit from linking culture to technology through ethnography and computation – which existed long before computers. An approach to Digital Craftsmanship could consider the ways in which human interactions are crafted through computation and making as a form of inquiry – re-conceptualizing “digital” instead as something computational, exploring the ways in which craft cultures are already computational and “digital.”



Bailey, Albert. 2013. Interview by Vernelle Noel, Personal Interview.

Blessing, Lucienne. 1995. “Comparison of Design Models Proposed in Prescriptive Literature.” Social Shaping of Design: The Role of Design in the Shaping of Technology. Commission of the European Communities, 187–212.

Brown, Ernest D. 1990. “Carnival, Calypso, and Steelband in Trinidad.” The Black Perspective in Music 18 (1/2): 81–100. doi:10.2307/1214859.

Dr. Jo-anne Tull. 2005. “Money Matters – Trinidad and Tobago Carnival 2005.” In . Trinidad and Tobago.

Eglash, Ron, Audrey Bennett, Casey O’donnell, Sybillyn Jennings, and Margaret Cintorino. 2006. “Culturally Situated Design Tools: Ethnocomputing from Field Site to Classroom.” American Anthropologist 108 (2): 347–62. doi:10.1525/aa.2006.108.2.347.

Kalisperis, L. 2002. “An Affordable Immersive Environment in Beginning Design Studio Education.” Thresholds – Design, Research, Education and Practice, in the Space Between the Physical and the Virtual [Proceedings of the 2002 Annual Conference of the Association for Computer Aided Design In Architecture / ISBN 1-880250-11-X] Pomona (California) 24-27 October 2002, Pp. 47-54.

Lee, Ann. 1991. “Class, Race, Colour and the Trinidad Carnival.” Social and Occupational Stratification in Contemporary Trinidad & Tobago, St Augustine: Institute of Social and Economic Research, 417–33.

“Lewicito ‘Cito’ Velasquez.” 2016. Accessed April 15.

Miller, Robert. 2013. Interview by Vernelle NoelPersonal Interview.

Noel, Vernelle A. A. 2013. “Trinidad Carnival:  Improving Design through Computation and Digital Technology.” Masters Thesis, Cambridge, Mass: Massachusetts Institute of Technology.

———. 2014. “The Bailey-Derek Wire Bending Grammar: A Shape Grammar Describing the Tacit Craft of Wire Bending in the Trinidad Carnival.”

Riggio, Milla Cozart. 2004. Carnival: Culture in Action – The Trinidad Experience. Routledge.

Ryan, Selwyn D, and Institute of Social and Economic Research. 1991. “Social and Occupational Stratification in Contemporary Trinidad and Tobago.” In . Institute of Social and Economic Research, University of the West Indies.

Schumacher, E. F. 1989. Small Is Beautiful: Economics as If People Mattered. New York: Harper Perennial.

TEDx Talks. 2016. The Power of Making: Craft, Computation, and Carnival. | Vernelle Noel Noel | TEDxPortofSpain. Accessed January 24.

Vernelle Noel. 2016. Vernelle A. A. Noel – H(EC)I: Wire-Bending and Design in the Trinidad Carnival. Accessed April 19.

Visser, Willemien. 2009. “Design: One, but in Different Forms.” Design Studies 30 (3): 187–223. doi:10.1016/j.destud.2008.11.004.



The Government of the Republic of Trinidad & Tobago, The Stuckeman Center for Design Computing, CARIRI, NEDCO, Felecia Davis, Daniel Cardoso Llach, David Goldberg, Xiao Han, Danielle Oprean, Shivaram Punathambekar, George Stiny, and the Stuckeman Family Building Community.

Contact or