http://www.chrisumbel.com/article/c_from_erlang_curl Comments about C from erlang via linked-in driver from the blog of Chris Umbel, software developer and database administrator. 2009-09-16 21:09:00 UTC http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c5bfa5c75fd7f3e43000008 Michael, it was actually another poster who mentioned another project using libcurl. I too am interested in knowing which project. Starting to look like I need threaded comments around here:) http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c5bfa5c75fd7f3e43000008 2010-08-06T12:04:44+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c5787cb75fd7f0f16000027 Hi, Michael, Well, I searched the solution in Internet and as a matter of course, I found it is provided erlang thread wrapper version of blocking C functions, as in GNU pth, State Threads Library, etc. http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c5787cb75fd7f0f16000027 2010-08-03T03:06:51+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c55aa3175fd7f7c0200002e Hi cignos, You note about blocking the OS thread is interesting. How can we fix Chris's C code then? http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c55aa3175fd7f7c0200002e 2010-08-01T17:09:05+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c55a99c75fd7f7c0200002c Hi Chris, You said: "I've heard of another project using libcurl with its erlang crawler". Which project is it? http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c55a99c75fd7f7c0200002c 2010-08-01T17:06:36+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002c It seems that C I/O functions block OS thread rather than erlang process. So, concurrent application of eurl:curl() seems to be serialized. Running ex1:run() with large concurreny (M) and slow respnding URL (Url) in following code illustrate this problem. ("curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);" in eurl.c kills Erlang VM on slow page due to timeout. So, running this test requires it disabled.) -module(ex1). -compile(export_all). run(M, N, Url) -> eurl:start(), spawn_testers(M, N, Url), receive _ -> ok end. spawn_testers(0, _, _) -> ok; spawn_testers(M, N, Url) -> spawn(?MODULE, tester, [N, Url]), spawn_testers(M - 1, N, Url). tester(N, Url) -> io:format("Start tester: ~p ~p~n", [now(), self()]), tester_loop(N, Url). tester_loop(0, _) -> ok; tester_loop(N, Url) -> eurl:curl(Url), io:format("~p: ~p ~p~n", [N, self(), now()]), tester_loop(N - 1, Url). http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002c 2010-07-23T08:07:41+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002b Oh, sure, no problem. I was just curious. As a linked-in driver example, it's great, and actually more interesting to me than others I've seen. http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002b 2010-06-24T00:06:13+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002a eee, it's been a while and have had little reason to look at the crawler in months but I dealt with some kind of persistent failure (I realize that's not useful information:)) that exhibited itself with extreme parallelization (which is why I went to erlang in the first place). I also do recall mucking with ibrowse and ultimately dismissing it while prototyping the project for some reason or another. Maybe over the next week or so I'll dig through the project and see if I can refresh my memory. Believe-you-me I'd love to keep that as pure-erlang as possible and, assuming it worked, I'd prefer a pure-erlang implementation. Also, keep in mind the purpose of the article. It's a description of how to write a linked-in driver in general, not http fetching. The curl-based http fetch was just an example payload even if it turns out to be contrived. http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf945313180400002a 2010-06-23T04:06:19+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000029 I've heard of another project using libcurl with its erlang crawler. Did you try ibrowse? Is there any theoretical benefit to libcurl over ibrowse (or any native erlang client, assuming it works)? How was the http module failing under pressure? http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000029 2010-06-22T00:06:46+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000028 Hey, thanks for the feedback. You're certainly right about that. Can't imagine I intended to leave it that way. I'm certainly cleaning that up. http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000028 2009-11-11T09:11:41+00:00 http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000027 Nice intro, except for the 'noop' thing which is pretty ugly actually :) The rest was a really nice intro though. /Mazen http://www.chrisumbel.com/article/c_from_erlang_curl#comment_4c4b2ecf9453131804000027 2009-11-06T09:11:54+00:00