Update

[less_retarded_wiki.git] / algorithm.md

# Algorithm

Algorithm (from the name of Persian mathematician Muhammad ibn Musa al-Khwarizmi) is an exact step-by-step description of how to solve some type of a problem. Algorithms are basically what [programming](programming.md) is all about: we tell [computers](computer.md), in very exact ways (with [programming languages](programming_language.md)), how to solve problems -- we write algorithms. But algorithms don't have to be just computer programs, they are simply exact instruction for solving problems. Although maybe not as obvious, [mathematics](math.md) is also a lot about creating algorithms because it strives to give us exact instructions for solving problems -- a mathematical formula usually tells us what we have to do to compute something, so in a way it is an algorithm too.

Cooking recipes are commonly given as an example of a non-computer algorithm, though they rarely contain branching ("if condition holds then do...") and loops ("while a condition holds do ..."), the key features of algorithms. The so called wall-follower is a simple algorithm to get out of any [maze](maze.md) which doesn't have any disconnected walls: you just pick either a left-hand or right-hand wall and then keep following it. Long division of numbers which students are taught at school is also an algorithm. You may write a crazy algorithm basically for any kind of problem, e.g. for how to clean a room or how to get a [girl](woman.md) to bed, but it has to be **precise** so that anyone can execute the algorithm just by blindly following the steps; if there is any ambiguity, it is not considered an algorithm; a vague, imprecise "hint" on how to find a solution (e.g. "the airport is somewhere in this general direction.") we rather call a [heuristic](heuristic.md). Heuristics are useful too and they may be utilized by an algorithm, e.g. to find a precise solution faster, but from programmer's point of view algorithms, the PRECISE ways of finding solutions, are the basics of everything.

Interesting fact: contrary to intuition there are problems that are mathematically proven to be unsolvable by any algorithm, see [undecidability](undecidability.md), but for most practically encountered problems we can write an algorithm (though for some problems even our best algorithms can be unusably [slow](time_complexity.md)).

Algorithms are mostly (possibly [not always](declarative.md), depending on exact definition of the term) written as a **series of steps** (or "instructions"); these steps may be specific actions (such as adding two numbers or drawing a pixel to the screen) or **conditional jumps** to other steps ("if condition X holds then jump to step N, otherwise continue"). At the lowest level ([machine code](machine_code.md), [assembly](assembly.md)) computers cannot do anything more complex than that: execute simple instructions (expressed as [1s and 0s](binary.md)) and perform conditional jumps -- in this computers are quite dumb (their strength comes from being able to execute many instruction with extreme speed). These jumps can be used to create **[branches](branch.md)** (in programming known as *if-then-else*) and **[loops](loop.md)**. Branches and loops are together known as [control structures](control_structure.md) -- they don't express a direct action but control which steps in the algorithm will follow. All in all, **any algorithm can be built just using these three basic constructs**:

- **sequence**: A series of steps, one after another. E.g. "write prompt, read number from input, multiply it by two, store it to memory".
- **selection** (branches, *if-then-else*): Two branches (blocks of instructions) preceded by a condition; the first branch is executed only if the condition holds, the second ("else") branch is executed only if the condition doesn't hold (e.g. "If user password is correct, log the user in, otherwise print out an error."). The second branch is optional (it may remain empty).
- **iteration** (loops, repetition): Sequence of steps that's repeated as long as certain condition holds (e.g. "As long as end of file is not reached, read and print out the next character from the file.").

Note: in a wider sense algorithms may be expressed in other (mathematically equivalent) ways than sequences of steps (non-[imperative](imperative.md) ways, see [declarative languages](declarative.md)), even mathematical equations are often called algorithms because they *imply* the steps towards solving a problem. But we'll stick to the common narrow meaning of algorithm given above.

Additional constructs can be introduced to make programming more comfortable, e.g. [subroutines/functions](function.md) (kind of small subprograms that the main program uses for solving the problem), [macros](macro.md) (shorthand commands that represent multiple commands) or [switch](switch.md) statements (selection but with more than two branches). Loops are also commonly divided into several types such as: counted loops, loops with condition and the beginning, loops with condition at the end and infinite loops (`for`, `while`, `do while` and `while (1)` in [C](c.md), respectively) -- in theory there can only be one generic type of loop but for convenience programming languages normally offer different "templates" for commonly used loops. Similarly to mathematical equations, algorithms make use of [variables](variable.md), i.e. values which can change and which have a specific name (such as *x* or *myVariable*).

Practical programming is based on expressing algorithms via [text](text.md), but visual programming is also possible: [flowcharts](flowchart.md) are a way of visually expressing algorithms, you have probably seen some. [Decision trees](decision_tree.md) are special cases of algorithms that have no loops, you have probably seen some too. Even though some languages (mostly educational such as [Snap](snap.md)) are visual and similar to flow charts, it is not practical to create big algorithms in this way -- serious programs are written as a text in [programming languages](programming_language.md).

## Example

Let's write a simple algorithm that counts the number of divisors of given number *x* and checks if the number is [prime](prime.md) along the way. (Note that we'll do it in a [naive](naive.md), educational way -- it can be done better). Let's start by writing the steps in plain [English](english.md) (sometimes called [pseudocode](pseudocode.md)):

1. Read the number *x* from the input.
2. Set the *divisor counter* to 0.
3. Set *currently checked number* to 1.
4. While *currently checked number* is lower or equal than *x*:
  - a: If *currently checked number* divides *x*, increase *divisor counter* by 1.
  - b: Increase *currently checked number*.
5. Write out the *divisor counter*.
6. If *divisor counter* is equal to 2, write out the number is a prime.

Notice that *x*, *divisor counter* and *currently checked number* are [variables](variable.md). Step 4 is a loop (iteration) and steps *a* and 6 are branches (selection). The flowchart of this algorithm is:

```
               START
                 |
                 V
               read x
                 |
                 V
       set divisor count to 0
                 |
                 V
       set checked number to 1
                 |
    .----------->|
    |            |
    |            V                no
    |    checked number <= x ? ------.
    |            |                   |
    |            | yes               |
    |            V                   |
    |     checked number    no       |
    |       divides x ? -------.     |
    |            |             |     |
    |            | yes         |     |
    |            V             |     |
    |     increase divisor     |     |
    |       count by 1         |     |
    |            |             |     |
    |            |             |     |
    |            |<------------'     |
    |            |                   |
    |            V                   |
    |     increase checked           V
    |       number by 1     print divisor count
    |            |                   |
    '------------'                   |
                                     V             no
                             divisor count = 2 ? -----.
                                     |                |
                                     | yes            |
                                     V                |
                           print "number is prime"    |
                                     |                |
                                     |<---------------'
                                     V
                                    END

```

This algorithm would be written in [Python](python.md) as:

```
x = int(input("enter a number: "))

divisors = 0

for i in range(1,x + 1):
  if x % i == 0: # i divides x?
    divisors = divisors + 1

print("divisors: " + str(divisors))
 
if divisors == 2:
  print("It is a prime!")
```

in [JavaScript](javascript.md) as:

```
function main()
{
  let x = parseInt(prompt("enter a number:"));
  let divisorCount = 0;

  for (let i = 1; i <= x; ++i)
    if (x % i == 0) // i divides x?
      divisorCount++;

  console.log("divisors: " + divisorCount);

  if (divisorCount == 2)
    console.log("It is a prime!");
}
```

in [C](c.md) as:

```
#include <stdio.h>
 
int main(void)
{
  int x, divisors = 0;

  printf("enter a number: ");
  scanf("%d",&x); // read a number

  for (int i = 1; i <= x; ++i)
    if (x % i == 0) // i divides x?
      divisors = divisors + 1;

  printf("number of divisors: %d\n",divisors);

  if (divisors == 2)
    puts("It is a prime!");

  return 0;
}
```

in [Forth](forth.md) as:

```
variable x
variable divisorCount 

: main
  ." enter a number: "
  0     \ number to read

  begin \ read the number by digits
    key

    dup dup 48 >= swap 57 <= and
    while

    48 - swap 10 * +
  repeat
  drop

  dup . cr
  x !
  0 divisorCount !

  x @ 1+ 1 do
    x @ i mod 0 = if \ i divides x?
      1 divisorCount +!
    then
  loop

  ." number of divisors: " divisorCount @ . cr

  divisorCount @ 2 = if
    ." It is a prime!" cr
  then
;

main
bye
```

in [comun](comun.md) as:

```
0
@@ # read X and convert to number
  <-
  $0 $0 "0" < >< "9" > | ? !@ .
  "0" - >< 10 * +
.
^

0 # divisor count
1 # checked number

@@
  $0 $3 > ?     # checked num. > x ?
    !@
  .

  $2 $1 % 0 = ? # checked num. divides x ?
    $1 ++ $:2   # increase divisor count
  .

  ++ # increase checked number
.

0 "divisors: " --> # write divisor count

$1 10 >= ?
  $1 10 / "0" + ->
.

$1 10 % "0" + -> 10 ->

$1 2 = ?
  0 "It is a prime" --> 10 ->
.
```

in Scheme [Lisp](lisp.md) as (here notice the difference in [paradigm](paradigm.md) -- loop is replaced with [recursion](recursion.md), as it's done in [functional](functional.md) programming):

```
(define (countDivisors x countFrom currentCount) ; recursive function
  (if (> countFrom x)
    (begin
      (display "divisor count: ")
      (display currentCount)
      (newline)
      (if (= currentCount 2)
        (display "It is a prime!\n")))
    (countDivisors x (1+ countFrom)
      (+ currentCount (if (zero? (remainder x countFrom)) 1 0)))))

(display "enter a number: ")
(countDivisors (read) 1 0)
```

and also for [fun](fun.md) in [Brainfuck](brainfuck.md) (a bit simplified version, made with our [Macrofucker](macrofucker.md) language):

```
[-]>[-]++++++++++++++++++++++++++++++++++++++++++++++++>,>[-]>[-]<<<[->>+>+<<<]
>>>[-<<<+>>>]<[-<->]<>[-]<[->+<]<[->+<]>>[-<<+>>]<[-]++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>[-]>
[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]<<-[->>>
[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<+>]<<<]>>[-<<<+>>>]<<<>[-]+++++++++++++++
+++++++++++++++++++++++++++++++++>,>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>
[-]<[->+<]<[->+<]>>[-<<+>>]<[-]++++++++++>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[
-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]<<-[->>>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>
]<[-<+>]<<<]>>[-<<<+>>>]<<<>[-]<[->+<]<[->+<]>>[-<<+>>]<<>[-]>[-]<<<[->>+>+<<<]
>>>[-<<<+>>>]<[-<+>]<>[-]++++++++++++++++++++++++++++++++++++++++++++++++>,>[-]
>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>[-]<[->+<]<[->+<]>>[-<<+>>]<<>[-]>[-]<<
<[->>+>+<<<]>>>[-<<<+>>>]<[-<+>]<>[-]<[->+<]<<<[->>>+<<<]>>>>[-<<<<+>>>>]<<<<>[
-]>[-]<<[->+>+<<]>>[-<<+>>]<[>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]>[-]<<<[->
>+>+<<<]>>>[-<<<+>>>]<>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<<<[-]>>>[-]>[-]>[-]<<
<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<+<[->
->[-]>[-]<<[->+>+<<]>>[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<<]<>[-]+<[[
-]>-<]>[<+>[-]]<[<<<+>>>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>[-]>[-]>[-]<
<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<+<[-
>->[-]>[-]<<[->+>+<<]>>[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<<]<>[-]+<[
[-]>-<]>[<+>[-]]<]<>[-]<[->+<]<[->+<]>>[-<<+>>]<<>[-]+<[[-]>-<]>[<+>[-]]<[<<<<+
>>>>[-]]<<-]<<>[-]+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]
>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<<<[-]>>>[-]
>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>
>>>]<+<[->->[-]>[-]<<[->+>+<<]>>[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<<
]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>[-
]>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+
>>>>]<+<[->->[-]>[-]<<[->+>+<<]>>[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<
<]<>[-]+<[[-]>-<]>[<+>[-]]<]<>[-]<[->+<]<[->+<]>>[-<<+>>]<<>[-]<[->+<]<[->+<]>>
[-<<+>>]<>[-]++++++++++++++++++++++++++++++++++++++++++++++++>[-]>[-]<<<[->>+>+
<<<]>>>[-<<<+>>>]<[-<+>]<.<<<[-]++++++++++>>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<
>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<<<[-]>>>[-]>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<
<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<+<[->->[-]>[-]<<[->+>+<<]>>
[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<<]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>
>>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>[-]>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-
<<<<+>>>>]<>[-]>[-]<<<<[->>>+>+<<<<]>>>>[-<<<<+>>>>]<+<[->->[-]>[-]<<[->+>+<<]>
>[-<<+>>]<>[-]+<[[-]>-<]>[<+>[-]]<[<<<+>>>[-]]<<]<>[-]+<[[-]>-<]>[<+>[-]]<]<>[-
]<[->+<]<[->+<]>>[-<<+>>]<<>[-]<[->+<]<[->+<]>>[-<<+>>]<>[-]+++++++++++++++++++
+++++++++++++++++++++++++++++>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<+>]<.<>[-]<
[->+<]<[->+<]>>[-<<+>>]<>[-]++++++++++++++++++++++++++++++++++++++++++++++++>[-
]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<+>]<.<<<<>[-]++++++++++.[-]++>[-]>[-]<<<[->
>+>+<<<]>>>[-<<<+>>>]<>[-]>[-]<<<[->>+>+<<<]>>>[-<<<+>>>]<[-<->]<>[-]+<[[-]>-<]
>[<+>[-]]<[[-]+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++.[-]++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++.[-]++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++.[-]++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++.[-]++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++.[-]++++++++++.[-]]
```

TODO: assembly, haskell, unix shell, ...

This algorithm is however not very efficient and could be **[optimized](optimization.md)** -- for example not only we wouldn't have to check if a number is divisible by 1 and itself (as every number is), but there is also no need to check divisors higher than the [square root](sqrt.md) of the checked value (mathematically above square root there only remain divisors that are paired with the ones already found below the square root) so we could lower the number of the loop iterations and so make the algorithm finish faster. You may try to improve the algorithm this way as an exercise :-)

## Study of Algorithms

Algorithms are the core topic of [computer science](compsci.md), there's a lot of theory and knowledge about them.

[Turing machine](turing_machine.md), a kind of mathematical bare-minimum computer, created by [Alan Turing](turing.md), is the traditional formal tool for studying algorithms, though many other [models of computation](model_of_computation.md) exist -- for example [lambda calculus](lambda_calculus.md) that's a basis of [functional programming](functional.md) under which we already see algorithms in a bit different light: not as a series of steps but rather as evaluating mathematical functions. From theoretical computer science we know not all problems are [computable](computability.md), i.e. there are problems unsolvable by any algorithm (e.g. the [halting problem](halting_problem.md)). [Computational complexity](computational_complexity.md) is a theoretical study of resource consumption by algorithms, i.e. how fast and memory efficient algorithms are (see e.g. [P vs NP](p_vs_np.md)). [Mathematical programming](mathematical_programming.md) is concerned, besides others, with optimizing algorithms so that their time and/or space complexity is as low as possible which gives rise to algorithm design methods such as [dynamic programming](dynamic_programming.md) (practical [optimization](optimization.md) is a more pragmatic approach to making algorithms more efficient). [Formal verification](formal_verification.md) is a field that tries to mathematically (and sometimes automatically) prove correctness of algorithms (this is needed for critical software, e.g. in planes or medicine). [Genetic programming](generic_programming.md) and some other methods of [artificial intelligence](ai.md) try to automatically create algorithms (*algorithms that create algorithms*). [Quantum computing](quantum.md) is concerned with creating new kinds of algorithms for quantum computers (a new type of still-in-research computers). [Programming language](programming_language.md) design is the art and science of creating languages that express computer algorithms well. Etcetc.

## Specific Algorithms

Following are some well known algorithms.

- [graphics](graphics.md)
  - [DDA](dda.md): line drawing algorithm
  - [discrete Fourier transform](fourier_transform.md): extremely important algorithm expressing signals in terms of frequencies
  - [Bresenham's algorithm](bresenham.md): another line drawing algorithm
  - [Midpoint algorithm](midpoint_algorithm.md): circle drawing algorithm
  - [flood fill](flood_fille.md): algorithm for coloring continuous areas
  - [FXAA](fxaa.md)
  - [Hough transform](hough_transform.md): finds shapes in pictures
  - [painter's algorithm](painters_algorithm.md)
  - [path tracing](path_tracing.md)
  - [ray tracing](ray_tracing.md)
  - ...
- [math](math.md)
  - [Boot'h algorithm](booths_algorithm.md): algorithm for multiplication
  - [Dijkstra's algorithm](dijkstras_algorithm.md)
  - [Euclidean algorithm](euclidean_algorithm.md): computes greatest common divisor
  - [numerical algorithms](numerical.md): approximate mathematical functions
  - [sieve of Eratosthenes](sieve_of_eratosthenes.md): computes [prime numbers](prime.md)
  - ...
- [sorting](sorting.md)
  - [bogosort](bogosort.md) (stupid sort)
  - [bubble sort](bubble_sort.md): simple, kind of slow but still usable sorting algorithm
  - [heap sort](heap_sort.md)
  - [insertion sort](insertion_sort.md)
  - [merge sort](merge_sort.md)
  - [shaker sort](shaker_sort.md)
  - [selection sort](selection_sort.md)
  - [slow sort](slow_sort.md)
  - [quick sort](quick_sort.md): one of the fastest sorting algorithms
  - ...
- [searching](searching.md)
  - [binary search](binary_search.md)
  - [linear search](linear_search.md)
  - ...
- [other](other.md)
  - [A*](a_start.md): path searching algorithm, used by [AI](ai.md) in many [games](game.md)
  - [backpropagation](backpropagation.md): training of [neural networks](neural_net.md)
  - [fizzbuzz](fizzbuzz.md): problem/simple algorithm given in job interviews to filter out complete [noobs](noob.md)
  - [FFT](fft.md): quickly converts signal (audio, image, ...) to its representation in frequencies, one of the most famous and important algorithms
  - [Huffman coding](huffman_code.md): [compression](compression.md) algorithm
  - [Kalman filter](kalman_filter.md)
  - [k-means](k_means.md): [clustering](clustering.md) algorithm
  - [MD5](md5.md): [hash](hash.md) function
  - [backtracking](backtracking.md)
  - [minimax](minimax.md) plus [alpha-beta pruning](alpha_beta.md): used by many [AI](ai.md)s that play turn based games
  - [proof of work](proof_of_work.md) algorithms: used by some [cryptocurrencies](crypto.md)
  - [RSA](rsa.md)
  - [Shor's algorithm](shors_algorithm.md): [quantum](quantum.md) factorization algorithm
  - [YouTube](youtube.md) algorithm: secret algorithm YouTube uses to suggest videos to viewers, a lot of people hate it :)
  - ...

## See Also

- [programming](programming.md)
- [design pattern](design_pattern.md)
- [recursion](recursion.md)