Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #54498 > unrolled thread

building an online judge to evaluate Python programs

Back to article view | Back to comp.lang.python

Contents

  building an online judge to evaluate Python programs Jabba Laci <jabba.laci@gmail.com> - 2013-09-20 19:28 +0200
    Re: building an online judge to evaluate Python programs Aseem Bansal <asmbansal2@gmail.com> - 2013-09-20 10:57 -0700
      Re: building an online judge to evaluate Python programs Jabba Laci <jabba.laci@gmail.com> - 2013-09-20 20:30 +0200
        Re: building an online judge to evaluate Python programs John Gordon <gordon@panix.com> - 2013-09-20 19:34 +0000
          Re: building an online judge to evaluate Python programs Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-09-22 12:53 -0400
    Re: building an online judge to evaluate Python programs John Gordon <gordon@panix.com> - 2013-09-20 20:08 +0000
      Re: building an online judge to evaluate Python programs Jabba Laci <jabba.laci@gmail.com> - 2013-09-21 00:26 +0200
      Re: building an online judge to evaluate Python programs Ned Batchelder <ned@nedbatchelder.com> - 2013-09-20 18:46 -0400
      Re: building an online judge to evaluate Python programs Ned Batchelder <ned@nedbatchelder.com> - 2013-09-21 16:29 -0400
      Re: building an online judge to evaluate Python programs Jabba Laci <jabba.laci@gmail.com> - 2013-09-21 21:57 +0200

#54498 — building an online judge to evaluate Python programs

Hi,

In our school I have an introductory Python course. I have collected a
large list of exercises for the students and I would like them to be
able to test their solutions with an online judge (
http://en.wikipedia.org/wiki/Online_judge ). At the moment I have a
very simple web application that is similar to Project Euler: you
provide the ID of the exercise and the output of the program, and it
tells you if it's correct or not. However, it can only be used with
programs that produce an output (usually a short string or a number).

In the next step I would like to do the following. The user can upload
his/her script, and the system tests it with various inputs and tells
you if it's OK or not (like checkio.org for instance). How to get
started with this?

There are several questions:
* What is someone sends an infinite loop? There should be a time limit.
* What is someone sends a malicious code? The script should be run in a sandbox.

All tips are appreciated.

Thanks,

Laszlo

[toc] | [next] | [standalone]

#54501

>However, it can only be used with programs that produce an output

Just interested, what else are you thinking of checking?

[toc] | [prev] | [next] | [standalone]

#54504

Let's take this simple exercise:

"Write a function that receives a list and decides whether the list is
sorted or not."

Here the output of the function is either True or False, so I cannot
test it with my current method.

Laszlo

On Fri, Sep 20, 2013 at 7:57 PM, Aseem Bansal <asmbansal2@gmail.com> wrote:
>>However, it can only be used with programs that produce an output
>
> Just interested, what else are you thinking of checking?
> --
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#54506

In <mailman.196.1379702349.18130.python-list@python.org> Jabba Laci <jabba.laci@gmail.com> writes:

> Let's take this simple exercise:

> "Write a function that receives a list and decides whether the list is
> sorted or not."

> Here the output of the function is either True or False, so I cannot
> test it with my current method.

Make a master input file and a master output file for each exercise.  If
the student program's output matches the master output when run from the
master input, then it is correct.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#54591

On Fri, 20 Sep 2013 19:34:02 +0000 (UTC), John Gordon <gordon@panix.com>
declaimed the following:

>In <mailman.196.1379702349.18130.python-list@python.org> Jabba Laci <jabba.laci@gmail.com> writes:
>
>> Let's take this simple exercise:
>
>> "Write a function that receives a list and decides whether the list is
>> sorted or not."
>
>> Here the output of the function is either True or False, so I cannot
>> test it with my current method.
>
>Make a master input file and a master output file for each exercise.  If
>the student program's output matches the master output when run from the
>master input, then it is correct.

	As long as the student doesn't have access to the master in/out data,
but only examples...

	Hearsay in my junior year at college was of a senior who couldn't
manage to get his program to work -- so he basically embedded lots of
output statements which basically wrote the expected output, based on
access to the test input data.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#54510

In <mailman.195.1379698177.18130.python-list@python.org> Jabba Laci <jabba.laci@gmail.com> writes:

> There are several questions:
> * What is someone sends an infinite loop? There should be a time limit.

You could run the judge as a background process, and kill it after ten
seconds if it hasn't finished.

> * What is someone sends a malicious code? The script should be run in a
> sandbox.

You could run the judge from its own account that doesn't have access to
anything else.  For extra security, make the judge program itself owned by
a separate account (but readable/executable by the judge account.)

I suppose you'd have to disable mail access from the judge account too.
Not sure how to easily do that.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#54516

I just found Docker ( http://docs.docker.io/en/latest/faq/ ). It seems
sandboxing could be done with this easily.

Laszlo

On Fri, Sep 20, 2013 at 10:08 PM, John Gordon <gordon@panix.com> wrote:
> In <mailman.195.1379698177.18130.python-list@python.org> Jabba Laci <jabba.laci@gmail.com> writes:
>
>> There are several questions:
>> * What is someone sends an infinite loop? There should be a time limit.
>
> You could run the judge as a background process, and kill it after ten
> seconds if it hasn't finished.
>
>> * What is someone sends a malicious code? The script should be run in a
>> sandbox.
>
> You could run the judge from its own account that doesn't have access to
> anything else.  For extra security, make the judge program itself owned by
> a separate account (but readable/executable by the judge account.)
>
> I suppose you'd have to disable mail access from the judge account too.
> Not sure how to easily do that.
>
> --
> John Gordon                   A is for Amy, who fell down the stairs
> gordon@panix.com              B is for Basil, assaulted by bears
>                                 -- Edward Gorey, "The Gashlycrumb Tinies"
>
> --
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#54520

On 9/20/13 6:26 PM, Jabba Laci wrote:
> I just found Docker ( http://docs.docker.io/en/latest/faq/ ). It seems
> sandboxing could be done with this easily.

At edX, I wrote CodeJail (https://github.com/edx/codejail) to use 
AppArmor to run Python securely.

For grading Python programs, we use a unit-test like series of 
challenges.  The student writes problems as functions (or classes), and 
we execute them with unit tests (not literally unittest, but a similar 
idea).  We also tokenize the code to check for simple things like, did 
you use a while loop when the requirement was to write a recursive 
function.  The grading code is not open-source, unfortunately, because 
it is part of the MIT courseware.

--Ned.
> Laszlo
>
> On Fri, Sep 20, 2013 at 10:08 PM, John Gordon <gordon@panix.com> wrote:
>> In <mailman.195.1379698177.18130.python-list@python.org> Jabba Laci <jabba.laci@gmail.com> writes:
>>
>>> There are several questions:
>>> * What is someone sends an infinite loop? There should be a time limit.
>> You could run the judge as a background process, and kill it after ten
>> seconds if it hasn't finished.
>>
>>> * What is someone sends a malicious code? The script should be run in a
>>> sandbox.
>> You could run the judge from its own account that doesn't have access to
>> anything else.  For extra security, make the judge program itself owned by
>> a separate account (but readable/executable by the judge account.)
>>
>> I suppose you'd have to disable mail access from the judge account too.
>> Not sure how to easily do that.
>>
>> --
>> John Gordon                   A is for Amy, who fell down the stairs
>> gordon@panix.com              B is for Basil, assaulted by bears
>>                                  -- Edward Gorey, "The Gashlycrumb Tinies"
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#54564

On 9/21/13 3:57 PM, Jabba Laci wrote:
> Hi Ned,
>
> Could you please post here your AppArmor profile for restricted Python scripts?
Laszlo, the instructions are in the README, including the AppArmor 
profile.  It isn't much:

#include <tunables/global>

<SANDENV>/bin/python {
     #include <abstractions/base>
     #include <abstractions/python>

     <SANDENV>/** mr,
     # If you have code that the sandbox must be able to access, add lines
     # pointing to those directories:
     /the/path/to/your/sandbox-packages/** r,

     /tmp/codejail-*/ rix,
     /tmp/codejail-*/** rix,
}

Note that there are other protections beyond AppArmor, setrlimits is also used to limit some resource use.

--Ned.

BTW: Top-posting makes it harder to follow threads of conversations, better form is to add your comments below the person you're replying to.

> Thanks,
>
> Laszlo
>
> On Sat, Sep 21, 2013 at 12:46 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:
>> On 9/20/13 6:26 PM, Jabba Laci wrote:
>>> I just found Docker ( http://docs.docker.io/en/latest/faq/ ). It seems
>>> sandboxing could be done with this easily.
>>
>> At edX, I wrote CodeJail (https://github.com/edx/codejail) to use AppArmor
>> to run Python securely.
>>
>> For grading Python programs, we use a unit-test like series of challenges.
>> The student writes problems as functions (or classes), and we execute them
>> with unit tests (not literally unittest, but a similar idea).  We also
>> tokenize the code to check for simple things like, did you use a while loop
>> when the requirement was to write a recursive function.  The grading code is
>> not open-source, unfortunately, because it is part of the MIT courseware.
>>
>> --Ned.
>>
>>> Laszlo
>>>
>>> On Fri, Sep 20, 2013 at 10:08 PM, John Gordon <gordon@panix.com> wrote:
>>>> In <mailman.195.1379698177.18130.python-list@python.org> Jabba Laci
>>>> <jabba.laci@gmail.com> writes:
>>>>
>>>>> There are several questions:
>>>>> * What is someone sends an infinite loop? There should be a time limit.
>>>> You could run the judge as a background process, and kill it after ten
>>>> seconds if it hasn't finished.
>>>>
>>>>> * What is someone sends a malicious code? The script should be run in a
>>>>> sandbox.
>>>> You could run the judge from its own account that doesn't have access to
>>>> anything else.  For extra security, make the judge program itself owned
>>>> by
>>>> a separate account (but readable/executable by the judge account.)
>>>>
>>>> I suppose you'd have to disable mail access from the judge account too.
>>>> Not sure how to easily do that.
>>>>
>>>> --
>>>> John Gordon                   A is for Amy, who fell down the stairs
>>>> gordon@panix.com              B is for Basil, assaulted by bears
>>>>                                   -- Edward Gorey, "The Gashlycrumb
>>>> Tinies"
>>>>
>>>> --
>>>> https://mail.python.org/mailman/listinfo/python-list
>>

[toc] | [prev] | [next] | [standalone]

#54565

Hi Ned,

Could you please post here your AppArmor profile for restricted Python scripts?

Thanks,

Laszlo

On Sat, Sep 21, 2013 at 12:46 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:
> On 9/20/13 6:26 PM, Jabba Laci wrote:
>>
>> I just found Docker ( http://docs.docker.io/en/latest/faq/ ). It seems
>> sandboxing could be done with this easily.
>
>
> At edX, I wrote CodeJail (https://github.com/edx/codejail) to use AppArmor
> to run Python securely.
>
> For grading Python programs, we use a unit-test like series of challenges.
> The student writes problems as functions (or classes), and we execute them
> with unit tests (not literally unittest, but a similar idea).  We also
> tokenize the code to check for simple things like, did you use a while loop
> when the requirement was to write a recursive function.  The grading code is
> not open-source, unfortunately, because it is part of the MIT courseware.
>
> --Ned.
>
>> Laszlo
>>
>> On Fri, Sep 20, 2013 at 10:08 PM, John Gordon <gordon@panix.com> wrote:
>>>
>>> In <mailman.195.1379698177.18130.python-list@python.org> Jabba Laci
>>> <jabba.laci@gmail.com> writes:
>>>
>>>> There are several questions:
>>>> * What is someone sends an infinite loop? There should be a time limit.
>>>
>>> You could run the judge as a background process, and kill it after ten
>>> seconds if it hasn't finished.
>>>
>>>> * What is someone sends a malicious code? The script should be run in a
>>>> sandbox.
>>>
>>> You could run the judge from its own account that doesn't have access to
>>> anything else.  For extra security, make the judge program itself owned
>>> by
>>> a separate account (but readable/executable by the judge account.)
>>>
>>> I suppose you'd have to disable mail access from the judge account too.
>>> Not sure how to easily do that.
>>>
>>> --
>>> John Gordon                   A is for Amy, who fell down the stairs
>>> gordon@panix.com              B is for Basil, assaulted by bears
>>>                                  -- Edward Gorey, "The Gashlycrumb
>>> Tinies"
>>>
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web