Hazard: May Cause Direct Harm#

A red diamond shaped outline (like a warning sign) with a skull in the middle.

Description#

The application area of this technology means that it is capable of causing direct physical or psychological harm to someone even if used correctly e.g. healthcare and driverless vehicles may be expected to directly harm someone unless they have 100% accuracy.

Examples#

Example 1: Software running on driverless cars can fail, allowing the car to crash, which can injure or kill people.

Example 2: AI chatbots, which are becoming more widely used in the medical field, may be used as medication reminders or for triaging services based on need.

Example 3: Recommender algorithms used on social media have been found to, within minutes, start recommending content about suicide, self-harm, or disordered eating to new accounts in recent research. This makes harmful content extremely accessible to young people.

Safety Precautions#

  • It is even more important that work of this nature is well-tested and that any “bugs” (mistakes in software) are found.

  • Carefully consider acceptable (and likely) margins of error - if no error is acceptable in this use case, then should it be done at all?