Jamie's Blog

Problem description, from Project Euler

Find the difference between the sum of the squares of the first N natural numbers and the square of the sum.

Solution

This one actually seemed to be easier than the others. No filters required!

    (defn problem06
        [upper]
        (defn sq [n] (* n n))
        (def l (range 1 (+ upper 1)))
        (-
            (sq (reduce + l))
            (reduce + (map sq l))
            )
        )

The biggest hangup I had was that when I originally read the problem, I thought I’d need a power function, and then spent at least 15 minutes determining there was no Clojure core power function. I found some alternatives here:

     ; for integers (cuz it's faster)
     (. (. java.math.BigInteger (valueOf 2)) (pow 5))
     ; for doubles
     (. java.lang.Math (pow 2 -3))

Problem description, from Project Euler

What is the smallest number that is evenly divisible by all of the numbers from 1 to N?

Solution

The math

Basically, we are finding the least common multiple (LCM) of the numbers (grade school math?). This can be done by generating the prime factors of each of the numbers. Once the factorization is complete, determine the greatest number of times a prime appears in any number. For example, the prime factors of 16 are (2,2,2,2) and of 20 are (2,2,5). 2 appears at most 4 times in numbers from 1-20, so we stick with (2,2,2,2). Repeat for all prime factors of the numbers 1 through 20.

An alternative way to think about this, is to list all the greatest powers of primes below 20, and find the product of these.
2^4 * 3^2 * 5 * 7 * 11 * 13 * 17 * 19. Doing it this way in the program would be quicker, but I decided to factor each number.

My first attempt

This solution is UGLY! For one, I hardcoded a list of primes. I do not yet know how to generate a list of primes quickly for arbitrary N, although I believe that will be required for future Euler problems (and Google will tell me how to do so).

Also, the structure of the program doesn’t seem truly functional to me. I may amend this post with a revision later. I did learn a number of great things in this including:

• How to flatten a sequence using tree-seq
• Working with hash maps

The biggest headache was the prime-factorization function, which I probably could’ve written in another language in 15-30 minutes. Instead, it took me maybe 6 hours to get the algorithm right, and I had to use a weird hack. I think I have some misunderstanding about the scope of a def call. I’m sure the topic will become illuminated as I proceed.

(defn problem05
    [upper]

    (defn flatten [x] (let [s? #(instance? clojure.lang.Sequential %)] (filter (complement s?) (tree-seq s? seq x)))) 

    ; cheated by using a list of primes. could be replaced by implementation of sieve of eratosthenes from problem 7
    ; http://www.math.utah.edu/~pa/math/primelist.html
    (def prime-cache [1 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 67 71 73 79 83 89 97 101 103 107 109 113 127 131 137 139 149 151 157 163 167 173 179 181 191 193 197 199 211 223 227 229 233 239 241 251 257 263 269 271 277 281 283 293 307 311 313 317 331 337 347 349 353 359 367 373 379 383 389 397 401 409 419 421 431 433 439 443 449 457 461 463 467 479 487 491 499 503 509 521 523 541 547 557 563 569 571 577 587 593 599 601 607 613 617 619 631 641 643 647 653 659 661 673 677 683 691 701 709 719 727 733 739 743 751 757 761 769 773 787 797 809 811 821 823 827 829 839 853 857 859 863 877 881 883 887 907 911 919 929 937 941 947 953 967 971 977 983 991 997 1009 1013 1019 1021 1031 1033 1039 1049 1051 1061 1063 1069 1087 1091 1093 1097 1103 1109 1117 1123 1129 1151 1153 1163 1171 1181 1187 1193 1201 1213 1217 1223 1229 1231 1237 1249 1259 1277 1279 1283 1289 1291 1297 1301 1303 1307 1319 1321 1327 1361 1367 1373 1381 1399 1409 1423 1427 1429 1433 1439 1447 1451 1453 1459 1471 1481 1483 1487 1489 1493 1499 1511 1523 1531 1543 1549 1553 1559 1567 1571 1579 1583 1597 1601 1607 1609 1613 1619 1621 1627 1637 1657 1663 1667 1669 1693 1697 1699 1709 1721 1723 1733 1741 1747 1753 1759 1777 1783 1787 1789 1801 1811 1823 1831 1847 1861 1867 1871 1873 1877 1879 1889 1901 1907 1913 1931 1933 1949 1951 1973 1979 1987 1993 1997 1999 2003 2011 2017 2027 2029 2039 2053 2063 2069 2081 2083 2087 2089 2099 2111 2113 2129 2131 2137 2141 2143 2153 2161 2179 2203 2207 2213 2221 2237 2239 2243 2251 2267 2269 2273 2281 2287 2293 2297 2309 2311 2333 2339 2341 2347 2351 2357 2371 2377 2381 2383 2389 2393 2399 2411 2417 2423 2437 2441 2447 2459 2467 2473 2477 2503 2521 2531 2539 2543 2549 2551 2557 2579 2591 2593 2609 2617 2621 2633 2647 2657 2659 2663 2671 2677 2683 2687 2689 2693 2699 2707 2711 2713 2719 2729 2731 2741 2749 2753 2767 2777 2789 2791 2797 2801 2803 2819 2833 2837 2843 2851 2857 2861 2879 2887 2897 2903 2909 2917 2927 2939 2953 2957 2963 2969 2971 2999 3001 3011 3019 3023 3037 3041 3049 3061 3067 3079 3083 3089 3109 3119 3121 3137 3163 3167 3169 3181 3187 3191 3203 3209 3217 3221 3229 3251 3253 3257 3259 3271 3299 3301 3307 3313 3319 3323 3329 3331 3343 3347 3359 3361 3371 3373 3389 3391 3407 3413 3433 3449 3457 3461 3463 3467 3469 3491 3499 3511 3517 3527 3529 3533 3539 3541 3547 3557 3559 3571 3581 3583 3593 3607 3613 3617 3623 3631 3637 3643 3659 3671 3673 3677 3691 3697 3701 3709 3719 3727 3733 3739 3761 3767 3769 3779 3793 3797 3803 3821 3823 3833 3847 3851 3853 3863 3877 3881 3889 3907 3911 3917 3919 3923 3929 3931 3943 3947 3967 3989 4001 4003 4007 4013 4019 4021 4027 4049])

    ; return primes below a certain number
    (defn primes
        [max-prime]
        ; we should actually have code to add to
        ; the prime cache, but it's not necessary
        ; to solve the problem
        (filter #(< % max-prime) prime-cache)
        )
    ; determines whether or not a number is a prime number
    (defn prime?
        [n]
        (not (nil? (some #(= n %) prime-cache)))
        )
    (defn prime-factorization
        [n]
        (defn pf-helper
            []
            (def first-divisible
                (first (filter
                    #(zero?
                        (rem n %)
                        )
                    (rest (primes n)))
                    )
                )
            ;(print "first-divisible: " first-divisible "\n")

            (def result (/ n first-divisible))
            ;(print "result: " result "\n")

            (def new-result
                (if (not (prime? result))
                    (prime-factorization result)
                    [result]
                    )
                )
            ;(print "new-result:" new-result "\n")
            ; i don't know why I need to set this again.
            ; the recursion seems to mutate a value that
            ; should not be in its scope
            (def first-divisible
                (first (filter
                    #(zero?
                        (rem n %)
                        )
                    (rest (primes n)))
                    )
                )
            (if (nil? new-result)
                nil
                (if (nil? first-divisible)
                    [new-result]
                    (flatten [new-result first-divisible])
                    )
                )
            )

        (if (prime? n)
            [n]
            (pf-helper)
            )
        )

    ; will return a hash where each list member is
    ; a key and the values are the count of the occurences
    ; of the member in the list
    (defn cnt-hash
        [haystack]
        (if (nil? haystack)
            {}
            ; increment the value in the hash
            (
                ; tail recursion?
                (def store (cnt-hash (rest haystack)))
                (def needle (first haystack))
                (assoc store needle
                    (if (contains? store needle)
                        (inc (get store needle))
                        1
                        )
                    )
                )
            )
        )
    ; looks at each list and returns the maximum
    ; number of occurences of each element in each
    ; list
    (defn list-max-occurences
        [l]
        (defn helper-reduce
            [h1 h2]
            (merge-with
                ; merges and returns the larger value
                ; if there is a key conflict
                (fn [a b] (if (> a b) a b))
                h1
                h2
                )
            )
        (reduce
            helper-reduce
            (map cnt-hash l)
            )
        )

    (def occurences
        (list-max-occurences
            ; returns a list of lists of prime-factors
            ; of the numbers between 1 and upper
            (map prime-factorization (range 1 (+ 1 upper)))
            )
        )

    ; returns the result of multiplying each key with its value
    (def products
        (map
            (fn [k] (. (. java.math.BigInteger (valueOf k)) (pow (get occurences k)))) ;(. java.lang.Math (pow k (get occurences k)))); (* k (get occurences k)))
            (keys occurences)
            )
        )
    (reduce * products)
)

I’m still not using the most current version of Clojure, so I may be rewriting functions which already exist.

Problem 2 Description, from Project Euler

Find the sum of all the even-valued terms in the Fibonacci sequence which do not exceed four million.

Solutions

My first attempt

The trickiest thing for me was figuring out how to write the fibonacci function in an elegant way. I know that with functional languages, recursion is supposed to be super elegant, and my solution was anything but.

    (defn problem02
        []
        (defn fib
            [termCount]
            (defn fib-help
                [termCount, terms]
                (def rterms (reverse terms))
                (def lst (first rterms))
                (def nlst (first (rest rterms)))
                (if (< (count terms) termCount)
                    (fib-help
                        termCount
                        (reverse (cons (+ lst nlst) rterms))
                        )
                    terms
                    )
                )
            (fib-help termCount, `(1, 2))
            )
        (reduce + (filter
            #(and
                (< % 4000000)
                (even? %)
                )
            (fib 200)
            ))

    )
    (problem02)

A solution by another person

The [site I linked to in the previous article](http://grok-code.com/367/learning-clojure-with-project-euler/) as a much more elegant solution. I really had trouble figuring out how to do an elegant Fibonacci algorithm, although I knew that mine was inelegant!

   (defn e2 [limit]
     (let [fibs2 (lazy-cat [0 1]
                           (map + fibs2 (rest fibs2)))]
       (reduce + (filter #(zero? (mod % 2))
                         (take-while #(< % limit) fibs2)))))

I love the user of *lazy-cat*. I was trying to search for how to do this, but wasn’t sure what this was called. I only knew that the *iterate* function used something like this (I guess I could’ve checked it’s source). I find the use of map interesting, but confusing because it seems as if *fibs2* would return a list and that there should somehow be duplicate values in that list. I’m sure I will come to understand more later.

After PhillyETE, I wanted to try out some Clojure, so I decided to solve the first problem of Project Euler. I coded up my own solution using some of the references on the Clojure site.

Project Euler Problem Description, from Project Euler

Find the sum of all the multiples of 3 or 5 below N.
(N is 1000 is this Project Euler problem.)

Solutions

My first attempt

    (def nmbrs (range 0 1000))
    (def multiples
        (filter
            #(and
                (not= % 0)
                (or
                    (=
                        (rem % 3)
                        0
                        )
                    (=
                        (rem % 5)
                        0
                        )
                    )
                )
            nmbrs
            )
        )
    (reduce + multiples)

Another attempt by a different author

Afterwards, I came across a terser, more elegant solution on a site that seems to have had the same idea as I did, Learning Clojure with Project Euler. The site had a really elegant looking solution:

    ; http://grok-code.com/367/learning-clojure-with-project-euler/
    (defn problem01 [limit]
     (reduce + (filter #(or (zero? (mod % 3))
                            (zero? (mod % 5)))
                       (take (- limit 1) (iterate inc 1)))))
    (problem 1000)

My revised attempt

The use of take and iterate are really interesting. I haven’t encountered these “lazy” functions, and was surprised when I tried to run (iterate inc 1) in the REPL. I decided to modify my version with some of the things I learned from the version above, and make it more terse.

    (defn problem01
        [upper]
        (reduce +
            (filter
                #(or
                    (zero?(rem % 3))
                    (zero?(rem % 5))
                    )
                (range 1 upper)
                ))
        )

I’ll be working on some more of the problems in the coming days.

(PS. If you are wondering why I use rem instead of mod, it’s because I am using an older version of clojure with TextMate–I’ve yet to upgrade)

I used jQuery for the UI. I am a recent convert to jQuery, having mostly used Prototype + Scriptaculous.

The word list is embedded into the page script as a javascript array. On document ready, html is generated, which writes the first and last word to the page, and creates blank input boxes for the intermediate words.

There is a keyup event bound on each input box, which will determine if the word is correct. If it is, a css class will be added which shows a green underline underneath the box. Otherwise, a red underline will be shown.

Finally, there are buttons on the page which are created dynamically and provides hints or reveal all of the answers.

Word Jumble Filled

Search

The problem of generating the chain of clues is a simple search problem. In this case, depth-first search was used, because the algorithm would attempt path depth-wise and only explore another branch if the generated chain was not long enough.

Another tactic would have been to use a breadth first search. To use breadth-first search, we could have modified the regex pattern to find all words that differed from the base word by just one letter.

Using water as the base word, that regular expression looks something like: /([^w]ater|w[^a]ter|wa[^t]er|wat[^e]r|wate[^r])/. This would find all words in the dictionary that differed by one word (let’s call this word set B).

If we were using breadth-first search, we would then repeat the process with all of the words we just found (word set B).

If you were to visualize the difference between breadth-first and depth-first search, breadth-first would look like a tree with wide but shallow roots. Depth-first search would look like a tree with few but deep roots.

Query Params

The flexibility of the puzzle is enhanced by optional query parameters that may be applied. The word param allows specification of the starting or seed word. The length param specifies the maximum length of the puzzle.

Recursion

The program uses recursion to perform the search. This almost goes without saying, for it is difficult to do general search without recursion (although you could do so with macros and similar programming constructs). Search may be done using loop control structures but I can’t imagine an elegant solution using loops.

The pseudocode for the recursion is basically:

function build(baseWord, chainWords, maxLength)

    regex = generateRandomRegex(baseWord)
    wordSetB = getPossibleWords(regex, notIn=chainWords)
    for(word in wordSetB)

        chain = build(word, chainWords+word, maxLength)
        if Length(chain) >= maxLength

            break

    return chain

I’m posting this because I always spend half-an-hour trying to find this information. To set up virtual mappings or virtual directories (what I call them) for ColdFusion on JRun on OSX, edit the jrun-web.xml file in (this is where mine was, you may have to look elsewhere):

/Applications/JRun4/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/jrun-web.xml

Add an entry to the jrun-web-app xml element like this…

<jrun-web-app>
<!-- ... -->

  <virtual-mapping>
    <resource-path>/virtualdir/*</resource-path>
    <system-path>/physical/path/</system-path>
  </virtual-mapping>

<!-- ... -->
</jrun-web-app>

The first thing I did was made sure that the word list would be cached on application start. This was as simple as creating an Application.cfc cfcomponent and implementing the onApplicationStart function.  This function reads the dictionary in (described in the last entry) and caches the word list in a ColdFusion array. There are other options for storing this data, but this had the best mix of speed and function considering the method of search I wanted to use against it.

Although the dictionary was only 52K, this caching probably helped performance a great deal.

To generate the word list, I decided on the following algorithm:

1. Choose an initial starting word (at random, or via user entry)
2. Use the word to generate a regular expression.
   Replace a random single letter with the Regex pattern [^L] (where L is the letter you have replaced).

   Example:

   word: water
   regex: w[^a]ter

3. Next, iterate through all of the words, testing each word against the regular expression. Store all matches.
4. With each match, one-by-one, repeat Step 2 until we get a chain of N words. (Where N is the maximum length of the chain.)
5. Obviously, if we have no more matches, we stop. If we have at least a 3-word chain, we can use it.

There are a few considerations not discussed above in generating the puzzle:

• If we match a word that is already in the chain, we should ignore that word to avoid duplicates.
• Not implemented: we should not replace a letter in the same position twice. For example, if we replace the “w” in water, don’t replace the “h” hater (if hater is the 2nd word).
• Depth-first versus Breadth-first searching…to be discussed

In my last entry, I described the concept behind the Word Jumble game. In this entry, I will describe initial steps in creating the game.

Firstly, I needed some dictionary of words. The Unix flavors have built-in dictionaries, and I develop on OSX, so I Googled the location of its dictionary:

/usr/share/dict/words

I knew I wanted to do puzzles of only 5-letter words, so I used the

grep

command to create a file of just these words.

grep ^.....$ /usr/share/dict/words > dictionary-5letterwords.txt

Notice the regular expression I used. I wanted to demonstrate an actual use of regular expressions for this project. The regular expression

/^.....$/

says to match a line of just 5 characters. The period means to match any character. I made the assumption that there would be no words in the dictionary with a space or other punctuation–although that was, perhaps, a faulty assumption.

Next, I started working on the code. Since we use mostly ColdFusion at Wharton, that’s what I wrote the app in.

Word Jumble Filled

For a recent Regular Expressions Tech Talk at Wharton, I wrote a Word Jumble game.  I will be describing the game and some of the key concepts used in making the game.

Game Screenshot

The premise of the game is to transform one word into another by replacing a single letter in the starting word to form a new word and repeating until you match the last word.

So, let’s say we have the word baked, which we want to transform into the word water. We do this by changing one letter in the word baked and forming a new word, repeating the single-letter replacement until we have the word water.

• baked <- start
• bated
• batea
• bater
• water <- end

The rules are:

• You are only given a starting word and ending word
• Change only one letter to form a new word
• The new word must be a real word (in some dictionary)
• The same word cannot appear twice
• Each word must be the same length

Based on these rules, I sat down one evening and spit out an implementation of the game which has the following features:

• randomly generates a puzzle of 5-letter words
• generates a puzzle of max length n, where n is the number of words in the puzzle
• accepts an initial starting word
• accepts user guesses for the intermediate words