- Last year, Steven Livingston, professor of political
communication at George Washington University, astonished attendees at
a conference on the geopolitical pros and cons of satellite imagery. He
didn't produce evidence of new military mobilizations or global pandemics.
Instead, he showed a video of figure skater Katarina Witt during a 1998
skating competition.
-
- In the clip, Witt gracefully plies the ice for about
20 seconds. Then came what is perhaps one of the most unusual sports replays
ever seen. The background was the same, the camera movements were the same.
In fact, the image was identical to the original in all ways except for
a rather important one: Witt had disappeared, along with all signs of her,
such as shadows or plumes of ice flying from her skates. In their place
was exactly what you would expect if Witt had never been there to begin
with - the ice, the walls of the rink and the crowd.
-
- So what's the big deal, you ask. After all, Stalin's
staff routinely airbrushed persona non grata out of photos more than a
half-century ago. And Woody Allen ushered a variation on reality morphing
into the movies 17 years ago with Zelig, in which he inserted himself next
to Adolf Hitler and Babe Ruth. In films such as Forrest Gump and Wag the
Dog, reality twisting has become commonplace.
-
- What sets the Witt demo apart - way apart - is that the
technology used to "virtually delete" the skater can now be applied
in real time, live, even as a camera records a scene and instantly broadcasts
it to viewers. In the fraction of a second between video frames, any person
or object moving in the foreground can be edited out, and objects that
aren't there can be edited in and made to look real. "Pixel plasticity,"
Livingston calls it. The implication for those at the satellite imagery
conference was sobering: Pictures from orbit may not necessarily be what
the satellite's electronic camera actually recorded.
-
- But the ramifications of this new technology reach beyond
satellite imagery. As live electronic manipulation becomes practical, the
credibility of all video will become just as suspect as Soviet Cold War
photos. The problem stems from the nature of modern video. Live or not,
it is made of pixels, and as Livingston says, pixels can be changed.
-
- The best-known examples of real-time video manipulation
so far are "virtual insertions" in professional sports broadcasts.
Last January 30, for instance, nearly one-sixth of humankind in more than
180 countries repeatedly saw an orange first-down line stretched across
the gridiron as they watched the Super Bowl. Princeton Video Imaging (PVI)
in Lawrenceville, N.J., created that line, stored it in a computer, and
inserted it into the live feed of the broadcast. To help determine where
to insert the orange pixels, several game cameras were fitted with sensors
that tracked the cameras spatial positions and zoom levels. Adding to the
illusion of reality was the ability of the PVI system to make sure that
players and referees occlude the virtual line when their bodies traverse
it.
-
- Last spring and summer, as PVI and rivals such as New
York-based Sportvision were airing virtual insertion products, including
simulated billboards on walls behind major league batters, a team of engineers
from Sarnoff Corp. in Princeton, N.J., flew to the Coalition Allied Operations
Center of NATO's Operation Allied Force in Vicenza, Italy. Their mission:
transform their experimental video processing technology into an operational
tool for rapidly locating and targeting Serbian military vehicles in Kosovo.
The project was dubbed TIGER, for "targeting by image georegistration."
"Our goal was to be able to fire precision-guided munitions at Serbian
military vehicles - just dial in the coordinates and the thing goes,"
explains Michael Hansen, a young, caffeinated Sarnoff gadgeteer who can
hardly believe he was helping fight a war last year.
-
- Compared to PVI's job, the military's technical task
was more difficult - and the stakes were much higher. Instead of altering
a football broadcast, the TIGER team manipulated a live video feed from
a Predator, an unmanned reconnaissance craft flying some 450 meters above
Kosovo battlefields. Rather than superimposing virtual lines or ads into
sports settings, the task was to overlay, in real time, "georegistered"
images of Kosovo onto the corresponding scenes streaming in live from the
Predator's video camera. The terrain images had been previously captured
with aerial photography and digitally stored. The TIGER system, which automatically
detected moving objects against the background, could almost instantly
feed to the targeting officers the coordinates for any piece of Serbian
hardware in the Predator's view. This was quite a technical feat, since
the Predator was moving and its angle of view was constantly changing,
yet those views had to be electronically aligned and registered with the
stored imagery in less than one-thirtieth of a second (to match the frame
rate of video recording). Any video that has ever been recorded is becoming
clip art that producers can digitally sculpt into the story they want to
tell.
-
- In principle, the targeting step could have been hotwired
to precision guided weapons. "We weren't actually doing that in Allied
Force," Hansen notes. "We were just telling targeting officers
exactly where Serbian targets were and then they would vector in planes
to go strike the targets." That way the human decision makers could
pre-empt flawed machine-made decisions. According to the Defense Advanced
Research Projects Agency, TIGER technology was used extensively in the
final three weeks of the Kosovo operation, during which "80 to 90
percent of the mobile targets were hit."
-
- So far, real-time video manipulation has been within
the grasp only of technologically sophisticated organizations such as TV
networks and the military. But developers of the technology say it's becoming
simple and cheap enough to spread everywhere. And that has some observers
wondering whether real-time video manipulation will erode public confidence
in live television images, even when aired by news outlets. "Seeing
may no longer be believing," says Norman Winarsky, corporate vice
president for information technology at Sarnoff. "You may not know
what to trust."
-
-
- The Sublime to the Ridiculous A crude form of video manipulation
already is happening in the satellite imagery community. The weekly publication
Space News reported earlier this year that the Indian government releases
imagery from its remote-sensing satellites only after defense facilities
have been "processed out." In this case, it's not real-time manipulation
and it's up front, like a censor's black marker. But pixels are plastic.
It is perfectly possible now to insert sets of pixels into satellite imagery
data that interpreters would view as battalions of tanks, or war planes,
or burial sites, or lines of refugees, or dead cows that activists claim
are victims of a biotech accident.
-
- A demo tape supplied by PVI bolsters the point in the
prosaic setting of a suburban parking lot. The scene appears ordinary except
for a disturbing feature: Amidst the SUVs and minivans are several parked
tanks and one armored behemoth rolling incongruously along. Imagine a tape
of virtual Pakistani tanks rolling over the border into India pitched to
news outlets as authentic, and you get a feel for the kind of trouble that
deceptive imagery could stir up.
-
- Commercial suppliers of virtual insertion services are
too focused on new marketing opportunities to worry much about geopolitics.
They have their eyes on far more lucrative markets. Suddenly those large
stretches of programming between commercials - the actual show, that is&emdash;become
available for billions of dollars worth of primetime advertising. PVI's
demo tape, for instance, includes a scene in which a Microsoft Windows
box appears&emdash;virtually, of course&emdash;on the shelf of
Frasier Crane's studio. This kind of product placement could become more
and more important as new video recording technologies such as TiVo and
RePlayTV give viewers more power to edit out commercials.
-
- Dennis Wilkinson, a Porsche-driving, sports-loving marketing
expert who became CEO of 10-year-old PVI about a year ago, couldn't be
happier about that. Wilkinson's eyes gleam when he describes a (near) future
in which virtual insertion technology pushes advertisements to the personalized
extreme. Combined with data-mining services by which browsers' individual
likes, dislikes and purchasing patterns can be relentlessly tracked and
analyzed, virtual insertion opens up the ability to shunt personally targeted
advertisements over phone lines or cables to Web users and TV viewers.
Say you like Pepsi but your neighbor next door likes Coke and your neighbor
across the street likes Seven-Up&emdash;the kind of data harvestable
from supermarket checkout records. It will become possible to tailor the
soft-drink image in the broadcast signal to reach each of you with your
preferred brand.
-
- Just 15 minutes up the road from PVI, Sarnoff's Winarsky
is also glowing&emdash;not so much about capturing market share as
about the transforming power of the technology. Sarnoff has a distinguished
history in that regard; the company is the descendant of RCA Laboratories,
which started innovating in television technology in the early 1940s and
has given birth to a plethora of media technologies. The color TV picture
tube, liquid crystal displays and high-definition TV all came, at least
in part, from RCA qua Sarnoff, which has five technical Emmys in its lobby.
-
- The ability to manipulate video data in real time, he
says, has just as much potential as some of these forerunners. "Now
that you can alter video in real time, you have changed the world,"
he says. That may sound inflated, but after looking at the Katarina Witt
demo, Winarsky's talk of "changing the world" loses some of its
air of hyperbole.
-
- Deleting people or objects from live video, or inserting
prerecorded people or objects into live scenes, is only the beginning of
the deceptions becoming possible. Pretty much any piece of video that has
ever been recorded is becoming clip art that producers can digitally sculpt
into the story they want to tell, according to Eric Haseltine, senior vice
president for R&D at Walt Disney Imagineering in Glendale, Calif. With
additional video manipulation technologies, previously recorded actors
can be made to say and do things they have never actually done or said.
"You can have dead actors star again in entirely new movies,"
says Haseltine.
-
- Contemporary shots featuring footage of dead performers
have been around for several years. But the Hollywood illusion-craft that,
for example, inserted John Wayne into a TV commercial required painstaking,
frame-by-frame post-production work by skilled technicians. There's a big
difference now, says Haseltine: "What used to take an hour [per video
frame], now can be done in a sixtieth of a second." This dramatic
speed-up means that manipulation can be done in real time, on the fly,
as a camera records or broadcasts. Not only can John Wayne, Fred Astaire
or Saddam Hussein be virtually inserted into pre-produced ads, they could
be inserted into, say, a live broadcast of The Drew Carey Show.
-
- The combination of real-time, virtual insertion with
existing and emerging post-production techniques opens up a world of manipulative
opportunity. Consider Video Rewrite technology, which its developers at
the Interval Corp. and the University of California, Berkeley first demonstrated
publicly three years ago. With just a few minutes of video of someone talking,
their system captures and stores a set of video snapshots of the way that
a person's mouth-area looks and moves when saying different sets of sounds.
Drawing from the resulting library of "visemes" makes it possible
to depict the person seeming to say anything the producers dream up&emdash;including
utterances that the subject wouldn't be caught dead saying.
-
- In one test application, computer scientist Christoph
Bregler, now of Stanford University, and colleagues digitized two minutes
of public-domain footage of President John F. Kennedy speaking during the
Cuban missile crisis in 1962. Using the resulting viseme library, the researchers
created "animations" of Kennedy's mouth saying things he never
said, among them, "I never met Forrest Gump." With technology
like this, near-future political activists conceivably will be able to
orchestrate webcasts of their opponents saying things that might make Howard
Stern sound like a mensch.
-
- Haseltine believes video manipulation techniques will
quickly be carried to their logical extreme: "I can predict with absolute
certainty," he says, "that one person sitting at a computer will
be able to write a script, design characters, do the lighting and wardrobe,
do all of the acting and dialog, and post production, distribute it on
a broadband network, do all of this on a laptop&emdash;and viewers
won't know the difference."
-
-
- The End of Authenticity So far, the widely witnessed
applications of real-time video manipulation have been in benign arenas
like sports and entertainment. Already last year, however, the technology
began diffusing beyond these venues into applications that raised eyebrows.
Last fall, for instance, CBS hired PVI to virtually insert the network's
familiar logo all over New York City&emdash;on buildings, billboards,
fountains and other places-during broadcasts of the network's The Early
Show. The New York Times ran a front-page story in January raising questions
about the journalistic ethics of altering the appearance of what is really
there.
-
- The combination of real-time virtual insertion, cyber-puppeteering,
video rewriting and other video manipulation technologies with a mass-media
infrastructure that instantly delivers news video worldwide has some analysts
worried. "Imagine you are the government of a hypothetical country
that wants more international financial assistance," says George Washington
University's Livingston. "You might send video of a remote area with
people starving to death and it may never have happened," he says.
-
- Haseltine agrees. "I'm amazed that we have not seen
phony video," he says, before backpedaling a bit: "Maybe we have.
Who would know?"
-
- It's just the sort of scenario played out in the 1998
movie Wag the Dog, in which top presidential aides conspire with a Hollywood
producer to televise a virtually crafted war between the United States
and Albania to deflect attention from a budding Presidential scandal. Haseltine
and others wonder when reality will imitate art imitating reality.
-
- The importance of the issue will only intensify as the
technology becomes more accessible. What now typically requires an $80,000
box of electronics the size of a small refrigerator should soon be doable
with a palm-sized card (and ultimately a single chip) that fits inside
a commercial video recorder, according to Winarsky. "This will be
available to people in Circuit City," he says. Consumer gear for virtual
video insertion is likely to require a camcorder with a specialized image-processing
card or chip. This hardware will take signals from the camera's electronic
image sensors and convert them into a form that can be analyzed and manipulated
in a computer using appropriate software&emdash;much as photo editors
at newspapers use Adobe Photoshop and other programs to "clean up"
digital image files. A home user might, for instance, insert absent family
members into the latest reunion tape or remove strangers they would prefer
not to be in the scene&emdash;bringing Soviet-style historical revisions
right into the family den.
-
- Combine the potential erosion of faith in video authenticity
with the so-called "CNN effect" and the stage is set for deception
to move the world in new ways. Livingston describes the CNN effect as the
ability of mass media to go beyond merely reporting what is happening to
actually influencing decision-makers as they consider military, international
assistance and other national and international issues. "The CNN effect
is real," says James Currie, professor of political science at the
National Defense University at Fort McNair in Washington. "Every office
you go into at the Pentagon has CNN on." And that means, he says,
that a government, terrorist or advocacy group could set geopolitical events
in motion on the strength of a few hours' worth of credibility achieved
by distributing a snippet of well-doctored video. A government, terrorist
or advocacy group could set geopolitical events in motion with a snippet
of well-doctored video
-
- With experience as an army reservist, as a staffer with
a top-secret clearance on the Senate's Intelligence Committee, and as a
legislative liaison for the Secretary of the Army, Currie has seen governmental
decision-making and politicking up close. He is convinced that real-time
video manipulation will be, or already is, in the hands of the military
and intelligence communities. And while he has no evidence yet that any
government or nongovernment organization has deployed video manipulation
techniques, real-time or not, for political or military purposes, he has
no problem conjuring up disinformation scenarios. For example, he says,
consider the impact of a fabricated video that seemed to show Saddam Hussein
"pouring himself a Scotch and taking a big drink of it. You could
run it on Middle Eastern television and it would totally undermine his
credibility with Islamic audiences."
-
- For all the heavy breathing, however, some experts remain
unconvinced that real-time video manipulation poses a real threat, no matter
how good the technology gets. John Pike, an analyst of the intelligence
community for the Federation of American Scientists in Washington, D.C.,
says the credibility risks are simply too great for governments or serious
organizations to get caught attempting to spoof the public. And for the
organizations that would be willing to risk it, says Pike, the news folks&emdash;knowing
just what the technology can do&emdash;will become increasingly vigilant.
-
- "If some human rights organization popped up at
CNN with some video, particularly an organization they were not familiar
with, I would think that [CNN] would consider that radioactive," says
Pike. Same goes for nongovernmental organizations (NGOs). "No responsible
director of an established organization would authorize such a thing. And
they would fire on the spot anyone caught doing it. The stock-in-trade
of NGO policy organizations is that 'we tell the truth.'"
-
- Even cool heads like Pike, however, concede that the
media's fortress of skepticism has an Achilles heel: the Internet. "The
issue is not so much your ability to get fake video on CNN, but to get
it online," he says. That's because so much Internet content is unfiltered.
"This could play into the phenomenon in the news production process
where you would not replicate the original report, but you might report
that it was reported," says Pike. And that could cascade into a CNN
effect. "These are undoubtedly experiments that will be done,"
Pike says.
-
- The trouble is, says Livingston, it may only take a few
such experiments to forever make people question the authenticity of video.
That could have enormous repercussions for military, intelligence and news
operations. An ironic sociological consequence might emerge: a return to
heavier reliance on unmediated face-to-face communication. In the meantime,
though, there will undoubtedly be some interesting twists and turns as
pixels become ever more plastic.
-
- -- Ivan Amato is a correspondent for National Public
Radio and the author of Stuff: The Materials the World Is Made Of a chronicle
of cutting-edge research in materials science. http://www2.bc.edu/~okeefew/349/rfppixels.htm
|
