Physical Computing

Personal computers have evolved in an office environment in which you sit on your butt, moving only your fingers, entering and receiving information censored by your conscious mind. That is not your whole life, and probably not even the best part.
—Dan O’Sullivan and Tom Igoe, Physical Computing:
Sensing and Controlling the Physical World with Computers

Physical computing is about computers interacting with more of the physical world, and using more of our physical bodies to interact with computers. “GUI technology allows you to drag and drop, but it won’t notice if you twist and shout.”1 In this appendix I shall explain two key ideas in physical computing, switches and transducers, and then show several interfaces for switching and transducing with Scratch. But first I would communicate some sense of the bigger picture of physical computing, because I believe it is one of the most important aspects of technomancy, especially as it is transmitted uniquely via Technomancy 101; and because it is fundamental to the connection between technomancy and robomancy.

Although the physical computing movement per se began recently, it is closely related to cybernetic ideas dating back to the early- and mid-20th century. Recall the “sensation/action paradigm of computing” I mentioned in the introduction; following are some diagrams showing how physical computing is related to the idea of how entities (whether organisms, mechanisms, or ???) interact with their environments.

Physical computing diagram from Wikipedia, showing interaction as flows of sensor and actuator activity between an interactive system and the real world [3PC]

 

“The interactive device,” from Getting Started with Arduino by Massimo Banzi and Michael Shiloh. Although the cloud in this diagram is intended to represent the interactive device, comparing it to the previous diagram, you can either think of the human as the interactive system and the cloud as the real world, or you can think of the human as the real world and the cloud as the interactive system. This interchangeability is important. [3PC]

 

“The feedback loop,” reproduced from Radical Constructivism: A Way of Knowing and Learning (1995) by Ernst von Glasersfeld. This diagram is turned 90° with respect to the above diagrams: think of everything above the ORGANISM/ENVIRONMENT dividing line as human, and everything below it as the cloud—or vice versa. The key addition here is feedback, requiring two parts: the control signal, i.e., the “observed input condition,” and the comparator, which compares (via the sense organ, or sensor) the system’s actual output to its desired output (the “reference”) and adjusts the output (via the action organ, or actuator) accordingly to bring it in line with whatever is desired. This cybernetic model of feedback was anticipated by the biosemiotics of Jakob von Uexküll.

 

“Basic schema for Craik’s general model of the process of thought,” reproduced from Algorithmic Aesthetics: Computer Models for Criticism & Design in the Arts by George Stiny ad James Gips. Refers to The Nature of Explanation (1943) by Kenneth Craik. This model slightly predates (but also anticipates) cybernetics (the first true cybernetics book was published in 1948), so it is missing the feedback loop of the previous diagram, but the other elements are familiar: the eye or “receptor” is the sense organ or sensor; the hands or “effector” is the action organ or actuator; and the “processor” is the comparator. In the case of an interactive computer system, the receptor could be a video camera, the effector could be a motor, and the processor could be a software program running on a CPU, which makes the motor spin in response to something the system “sees.” Replace the video camera with bump and cliff sensors, and the motor with a pair of motorized wheels, and you have something like a Roomba—but that is taking us into robomancy territory!

Switches

In electrical engineering and electronics, a switch is a device that can interrupt the flow of electricity through a circuit or divert electricity from one part of the circuit to another. Perhaps the most common example is a light switch: in one state (“on”) it passes electricity through the lamp causing the latter to illume; in another state (“off”) it prevents electricity from reaching the lamp and so the lamp remains dark.

Lamp Switch Diagrams
[3PC (source)]

Digital computation involves switches in two ways. One is encoding. E.g., you could say that the switch being open (indicated by the lamp being off) represents logical falsity, and the switch being closed (indicated by the lamp being on) represents logical truth. If someone were to ask you a true-or-false (or yes-or-no) question, you could answer by switching the lamp on or off. All binary data are essentially encoded this way; e.g., the letter ‘A’ in the ASCII encoding scheme is represented by the binary value ‘01000001’, which you could express with eight switches (or one switch eight times): the first one open, the second one closed, the next five open, and the last one closed.

The other way digital computation uses switches is with logic gates, which output a value dependent on the state of one or (usually) more input values.2 E.g., a simple AND gate has two inputs and one output; if both inputs are ‘true’, the gate outputs ‘true’; if either input is ‘false’, the gate outputs ‘false’. An OR gate returns ‘true’ if either input is ‘true’, and only returns ‘false’ if both inputs are ‘false’. A NOT gate has but one input and returns ‘true’ if the input is ‘false’, and ‘false’ if the input ‘true’. Computers use combinations of logic gates to make decisions: if a and (b or c) and not d then e, &c. Computer programming statements that make choices between multiple possible states are like software switches, and some programming languages even include a switch statement that implements a specific kind of decision.

Computers and computer software allow us to make complex or unique switches more easily than by implementing such things in hardware alone.

Without a computer, you can connect a button being pressed to a light turning on. With a computer, you can make the relationship between the button and the light more complex. For example, you can make the light's turning on dependent on the number of times the button was pressed, for how long it was pressed, or whether it was pressed in conjunction with other buttons in other rooms or on other continents. You can change the relationships on the fly; for example, you can make the light come on after two button presses during the day, and only after one button press at night. To get the computer to make these relationships between events it senses and events it causes, you write computer programs.3

Switching allows us to program technomantic interactions that respond to gestures, sounds, the presence of objects (see “Electrical Materials”), or the actions performed with them. Examples of switches are observable in many Technomancy 101 projects. The AppleOfDiscord, e.g., functions quite like a light switch, but instead of a plastic toggle there is an enchanted apple, and instead of activating a lamp it selects a divinatory sign.

Transducers

Transducers convert variations in one physical quantity to variations in another. The word ‘transducer’ is from the Latin transducere: to lead, bring, transport, or conduct across or over something. Transduction always occurs across a boundary, such as when the eye receives light and converts it to electrical impulses that travel to the brain in the act of seeing, or when electrical signals from the brain activate motor neurons causing hand muscles to move a pencil along a sheet of paper in the act of drawing. Here the eye and hand are interfaces between the body and the space it inhabits—interfaces across which interactions take place.

Electronic sensors and actuators do the same thing. A photoresistor, e.g., converts light into electrical resistance, whereas a loudspeaker converts an electrical audio signal into a sound. Analog-to-digital converters convert analog electrical signals to digital signals a computer can understand.

Transducers may be used like switches; e.g., a photoresistor can activate (or deactivate) something when a certain amount of light is detected. Such is a binary interaction: on/off. But you can also use transducers in an analog way, which often involves mapping a value from one range to another. E.g., the light sensor (i.e., a photocell connected to an ADC in the PicoBoard (v.i.) returns a value between 0 (complete darkness) and 100 (brightest light the sensor can detect). Say you want to move a Scratch sprite across the screen as the light sensor’s signal changes, from the leftmost side of the screen in complete darkness, to all the way right in the brightest light. The Scratch screen area ranges from −240 to +240, for a total of 480 positions—not 100. You need to map the light sensor’s signal from a range of 1 to 100, to a range of −240 to 240. You can do this using the function:

f(x) = (x − input_start) / (input_end − input_start) * (output_end − output_start) + output_start

Where x is the light sensor value; input_start and input_end are 0 and 100, respectively; and output_start and output_end are −240 and 240 respectively. Here is a Scratch project I made that interactively illustrates the mapping function.

Interfaces

There are many devices that can connect to Scratch for the purpose of connecting Scratch to the physical world. Following are data about the hardware featured or mentioned in Technomancy 101. These may be combined with electrical materials for a great variety of technomantic designs.

Camera & Microphone

If you are running Scratch on a recently manufactured computer, there is a good chance that computer has a webcam with microphone built into it, or you can connect external ones. The Scratch Sensing palette includes blocks for using your camera and mic as sensors. Here are a few examples:

Camera and microphone scripts

Makey Makey

Makey Makey Classic

Makey Makey is a set of touch sensors that are interpreted as keyboard or mouse events by your computer. Since Scratch has blocks for responding to keyboard and mouse events, you can use Makey Makey to make Scratch react whenever you touch or connect something that is at least a little electrically conductive. It may be easier to understand what I mean by observing Makey Makey in action:

You can buy Makey Makey directly from its website, or SparkFun sometimes stocks it. Makey Makey is based on Arduino (v.i.), and can be hacked and reprogrammed using the Arduino IDE.

Makey Makey GO is a USB stick that can respond as a single keyboard or mouse event (the defaults are space-bar and right-click but those can be changed). Unlike the Makey Makey Classic, the GO does not require a ground connection, which makes it even easier to use. I like the GO so much I keep one in my sling with my other everyday tools.

Makey Makey GO

There are a few kinds of switches you can implement with Makey Makey. All of the following examples assume the object you are touching sends a space-bar signal to Scratch, except the fourth which assumes you are touching two objects: one interpreted as the ‘space’ key and the other as the ‘w’ key (requires Makey Makey Classic).

Makey Makey scripts

Makey Makey Classic allows for programming interactions where you connect objects to the computer and leave them connected for some time, by simultaneously attaching both the signal and ground wires to the object. E.g., you could make SIBOR function only when its material sigil, inscribed in electrically conductive paint, is connected to Makey Makey (you could accomplish this kind of interaction as well with a PicoBoard or Raspberry Pi).

You can use Makey Makey Classic to light an LED or power a small motor when a connected object is touched, without doing anything in Scratch.

Makey Makey resources:

PicoBoard

SparkFun PicoBoard

The PicoBoard was developed by the Playful Invention Company (PICO), and was based on an earlier apparatus called the Scratch Sensor Board. Today, SparkFun manufactures and sells this version of the PicoBoard; its functionality is the same. It includes a button that can be in one of two states (pressed or not pressed); a slider that changes values as it moves from one side to the other; light and sound sensors; and four electrical resistance sensors.

The PicoBoard works with Scratch 2.0 online or off-line versions, and Scratch 1.4 including on the Raspberry Pi (v.i.). Its design is open-source, so you can legally modify it or manufacture your own, and you can sometimes find modified versions online.

The resistance sensors each require a pair of crocodile clips that when connected to an object measure the electrical resistance of that object. This facilitates programming interactions involving:

Here are just a few examples of things you could do with the PicoBoard:

PicoBoard scripts

Makey Makey and PicoBoard can both be used to interact with objects made of electrically conductive materials, e.g., a sigil inscribed in conductive paint, but which one should you use?

Makey Makey is better for when you want something to happen when you touch the sigil, either with part of your body or with another conductive object connected to Makey Makey.

PicoBoard is superior when you want something to happen so long as the sigil is connected to the board. The latter can also be accomplished with Makey Makey Classic by attaching the sigil to signal and ground connections at the same time, but keep in mind that Makey Makey accomplishes this by repeatedly sending the same keyboard signal to Scratch, which may interfere if your project requires you to input text at some point; whereas PicoBoard has a dedicated reporter block within Scratch for returning the value of the sigil’s electrical resistance.

PicoBoard can also be used to detect temporary contact between two conductive objects, by attaching each object to one end of the pair of crocodile clips plugged into the board and detecting the resistance change when they touch, but it is not sensitive enough to detect you tapping an object with your finger.

PicoBoard resources:

Here are some Sparkfun products that work well with the PicoBoard (you can get many of these from Adafruit or other vendors, too):

LEGO WeDo

LEGO WeDo [3PC (source)]

LEGO WeDo is a lower-primary-school STEM education platform that includes a motor, distance/motion sensor, and tilt sensor, all compatible with the LEGO construction system. The kits are not inexpensive, but you can acquire the pieces individually from eBay and other markets. The motor (LEGO part #8883) is the same one found in the LEGO Power Functions set, and the PF lights (#8870) and extension cable (#8871) also work with WeDo. The extension cable will also allow you to drive a LEGO Technics 9v mini-motor (#71427/#43362; out of production but you can still find them for resale), or perhaps a hacked block for a custom circuit of your own design.

In early 2016, LEGO released WeDo 2.0, which has similar capabilities to the previous version but features a wireless (Bluetooth) controller with built-in RGB LED and a speaker that can play tones, and it can interface with all LEGO Power Function 2.0 sensors and actuators.

Scratch 2.0 and 1.4 include built-in extensions that add blocks for controlling the WeDo motors and reading data from the WeDo sensors. Scratch 2.0 includes an extension for WeDo 2.0, adding blocks to set the light color and play notes through the speaker

LEGO is highly modular and can be used to rapidly prototype a good variety of machines and interactions, but it is expensive compared to most of the other tech on this page. If the cost is prohibitive, do not worry, for anything you can make with WeDo you can make without it; it just might require a little more work to make it.

Following is a list of LEGO part numbers for compatible parts:

  • WeDo 1.0 kit: #9580
  • 1.0 USB hub: #9581
  • 1.0 motor: #8883 (works also with Technic motors #71427 and #43362)
  • 1.0 motion/distance sensor: #9583
  • 1.0 tilt sensor: #9584
  • 1.0 light: #8870
  • 1.0 extension wire: is 8″ (20cm) #8886, 20″ (50cm) #8871
  • WeDo 2.0 kit: #45300
  • 2.0 Smart Hub: #45301
  • 2.0 rechargeable battery: #45302
  • 2.0 motor: #45303
  • 2.0 motion/distance sensor: #45304
  • 2.0 tilt sensor: #45305

Arduino

Arduino UNO R3

Arduino is an open-source prototyping platform based on the Atmel AVR family of microcontrollers. There are many different Arduino boards; the one pictured above is the UNO, the most commonly used board (at the time I wrote this).

Arduino capabilities can be extended with boards call shields that plug into the Arduino’s pin headers.

Arduino is open-source hardware, so there are legitimate other boards based on its design, such as the SparkFun RedBoard and Adafruit Metro.

There are a few ways to interface Scratch with Arduino:

The Scratch Arduino extension for Scratch 2.0 no longer works in recent builds of Chrome or Firefox. I do not yet know if there will be a new extension for Scratch 3.0, but I will update this page if that happens.

Raspberry Pi

Raspberry Pi 3

The Raspberry Pi is an inexpensive, credit-card-sized, single-board computer that can be run in various configurations including desktop, laptop, video game console, media center, and embedded system. Raspbian, the Pi’s official operating system, is a variant of Debian Linux optimized for the Pi’s architecture. The current version of Raspbian includes special versions of Scratch 1.4 and 2.0 modified to interact with the Pi’s general-purpose input/output (GPIO) pins. The Pi’s Scratch 2.0 also includes an extension to interface with the Sense HAT (“hardware attached on-top,” similar to Arduino’s shields; you can also work with the Sense HAT in 1.4). Scratch 1.4 is able to interface with some older extension hardware for the Pi, including PiFace and Gertboard.

The Raspberry Pi differs from the other interfaces mentioned here in that it is a personal computer with the form factor of a sensor board or microcontroller. That means you can run Scratch on the Pi itself, but operate the Pi in headless or semi-headless mode, i.e., without a monitor, keyboard, or mouse attached to it. I use RealVNC for remote access to my Pis (I have several running in various places) from my laptop, tablet, or phone.

Raspberry Pi resources:

Others

Other Devices
The Emotiv EPOC and Insight neuroheadsets, Myo armband controller, and Leap Motion touchless controller can add high-tech interactions to your Scratch projects

Because Scratch can respond to keyboard and mouse input, any device that can appear to your computer as a mouse or keyboard, or which has a utility for mapping the device’s data to keyboard or mouse signals, can interface with Scratch (this is exactly what Makey Makey does, and the same example scripts for it apply to these other devices as well). This includes many video game controllers, the Leap Motion controller (which also has a third-party Scratch extension), the Myo gesture control armband, and the Emotiv EPOC and Insight brain-computer interfaces. These allow you to program Scratch to respond to your hand gestures, arm movements, thoughts or emotions, &c.

Wireless Input
Although physical computing tends to eschew keyboards, when keyboard input is necessary, wireless keyboards can free you from having to sit or stand in front of your computer. Game controllers and even presentation wands can be programmed to send keyboard or mouse signals, so you can use them to start or stop Scratch projects or select options while a project is running.
  1. Dan O’Sullivan and Tom Igoe, Physical Computing: Sensing and Controlling the Physical World with Computers (Thomson, 2004) xvii.
  2. This tutorial from Electronics for Dummies illustrates the relationship between switches and logic gates by showing how to wire two DPDT switches to make AND, OR, and NOT gates.
  3. O’Sullivan and Igoe, op. cit., xviii.