SocialBot Grand Challenge FAQs
Frequently asked questions about the challenge.
What new datasets will I have access to as part of the Alexa Prize?
Teams competing in the Alexa Prize will have access to the Extended Topical Chat dataset, which includes the results of on-going collections and annotations, in addition to the many other resources exclusive to Alexa Prize participants.
The dataset is a corpus of human-human social conversations collected from crowd workers. Each conversation (and each turn of the conversation) in this dataset is linked to knowledge provided to crowd workers. The knowledge is collected from a variety of unstructured or loosely structured text resources, and each conversation refers to a related set of entities. None of these conversations are interactions with Alexa customers.
Is any of this data reproduceable?
Yes. The Topical Chat dataset will be released publicly to the research community, to support the publication of high quality, repeatable research on September 17, 2019.
What is a SocialBot?
While definitions vary, in the context of the Alexa Prize, a socialbot is an Alexa skill that engages customers in a conversation about popular topics such as entertainment, sports, politics, technology, and fashion.
Can I choose to build any kind of conversational bot?
No, this competition focuses solely on socialbots. Your socialbot will need to converse about a wide range of popular societal topics and current events, and not only topics of your choosing. You will develop your own technologies relevant to your approach, such as natural language understanding, dialog management, knowledge acquisition, commonsense reasoning, conversational planning, machine learning, natural language generation, etc. Far-field automatic speech recognition (ASR) and text-to-speech (TTS) will be provided by Amazon, along with additional data, models and tools.
What will my SocialBot do?
Your socialbot will be an Alexa skill that can converse coherently and engagingly with humans on popular topics and current events. Alexa users in the U.S. will ask to converse with your socialbot about topics of interest to them, such as baseball playoffs, celebrity gossip, or scientific breakthroughs. For example:
- User: Let’s chat about the Mars Mission.
- Socialbot: There are multiple Mars missions, some public and some private.
- User: Who do you think will succeed?
- Socialbot: I think more than one will succeed. The first one to get to Mars will open the doors for others.
- User: I’d love to go to Mars.
- Socialbot: Me too. Luckily I can hitch a ride on any computer.
- User: That’s very funny.
- Socialbot: The biggest challenge isn’t technology, it’s funding. It will cost $500 billion to send humans to Mars.
Your socialbot will continue turn-by-turn interaction, starting with a topic the user asked for, until the user chooses to stop. Like an everyday human conversation, the interaction may shift naturally to related topics, as in the above example, but the conversation should remain coherent, relevant, and engaging. Your socialbot may suggest topics to keep the conversation flowing. The goal is to keep the conversation from deteriorating to the point where the user loses interest.
How will I build my SocialBot?
You will use the Alexa Skills Kit (ASK) to build an Alexa skill, hosted on AWS Lambda that will create the end-to-end conversational experience for a user. Using the provided APIs, your skill will receive as input the text of the user’s utterance, and produce as output a text sentence that will be spoken to the user. You do not need to tackle ASR (automatic speech recognition) or TTS (text to speech). You will also be provided with the CoBot Toolkit (a conversational bot toolkit), a software development kit that works with ASK and was built specifically for Alexa Prize teams to reduce the involved engineering in setting up a socialbot and allow teams to focus on the science.
Your skill will need to determine an appropriate response at each turn of the conversation. It will also need to keep up with current news and events using the provided data sources. You may use additional data sources or libraries if you wish, subject to the terms described in the Official Rules.
What is the Alexa Skills Kit (ASK)?
The Alexa Skills Kit (ASK) is a collection of free, self-service APIs, tools, documentation, and code samples that make it fast and easy for you to add skills to Alexa. Your team will use ASK to build, deploy, and test socialbots that are capable of conversing with millions of Alexa customers.
What is the goal of the challenge?
The goal of the Socialbot Grand challenge is to advance several areas of conversational AI including natural language understanding (NLU), context modeling, dialog management, commonsense reasoning, natural language generation (NLG), and knowledge acquisition. The grand challenge objective is to create a socialbot that converses coherently and engagingly with humans on popular topics for 20 minutes while achieving a customer rating of at least 4.0/5.0.
How will winners be selected?
Midway through the competition, a quarterfinal round will take place in which all socialbots that have been certified and published will be divided into two brackets randomly. An Amazon panel and Amazon customers will evaluate these socialbots, and the three socialbots selected by Amazon Alexa customers in each bracket, as well as at least one socialbot selected by the Amazon panel will advance to the semifinals.
During the semifinals round, an Amazon panel and Amazon customers will evaluate the semifinalist socialbots (and other socialbots that have elected to continue during the semifinals round). Three socialbots selected by Amazon Alexa customers and at least two socialbots selected by the Amazon panel will advance to the finals.
Will this competition be judged like a Turing Test?
No. The goal of the Alexa Prize is to create socialbots that engage in interesting, human-like conversations, not to make them indistinguishable from a human when compared side-by-side. While the socialbots built for the Alexa Prize will be human-like in some respects, they will be very different in others, and could easily reveal themselves in a Turing Test. For example, socialbots may have ready access to much more information than a human. Asking the socialbots to act human could diminish the customer experience and hinder the efforts of the participants to build the best socialbot to further conversational AI.
When and where is the final event?
The final judging will be held in May 2020 at a location to be determined, with the final results announced in June 2020.
Can we use other funding to help us participate in this challenge?
Yes, you may use other funding to support your team, subject to the terms described in the Official Rules. External funding will need to be disclosed by April 29, 2020.
Will Alexa customers be able to engage with our socialbot?
By October 10, 2019, your team will be required to submit its socialbot for certification and publication by the Amazon Alexa team. After certification, you will enter the Internal Amazon Beta Period, where Amazon employees will test your socialbot and provide feedback. After the Internal Amazon Beta Period, we will allow Amazon Alexa customers to try your socialbot and provide feedback to you. Amazon may impose Availability Criteria, or requirements the socialbots must meet before it will be made available to Alexa users. Availability Criteria may include criteria such as a minimum average customer rating, uptime requirements, or an ability to consistently filter offensive content.
Amazon launched the Echo in the UK, Germany, India, Japan, and other countries. Will localized languages be supported?
Your team must build its socialbot using U.S. English. Your socialbot will be available to Alexa customers in the U.S. Customers in other countries may also access it by setting their Amazon PFM (Preferred Marketplace) to U.S.
Will we publish our research from the Alexa Prize?
Yes. Publishing research papers as an outcome of your work on the Alexa Prize is required for all teams participating in the competition, although teams should not publish Amazon confidential information, as described in the Official Rules. The Alexa Prize requires all teams to submit a technical paper for the Alexa Prize proceedings. Your socialbot will not be selected for the finals if your team does not submit a technical paper for Alexa Prize proceedings. Papers will be published at the end of the competition in an online Proceedings of the Alexa Prize, which will be publicly available.
Teams may also publish research papers in third-party publications and conferences, as long as all papers are provided to Amazon for review at least two weeks before the submission deadlines and no research papers are published before the Alexa Prize proceedings are published without Amazon’s prior approval.
Who will own the intellectual property rights in my submission?
You will retain ownership over your socialbot. Amazon will have a non-exclusive license to any technology or software you develop in connection with the competition. See the Official Rules for details.
Who can apply to participate?
The Alexa Prize is open to full-time students enrolled in an accredited university, with the exception of universities in Cuba, Iran, North Korea, Sudan, Syria, and the region of Crimea (see Official Rules). Proof of enrollment will be required to participate.
Can I participate if I don’t attend a university?
No. The Alexa Prize is open only to full-time enrolled university students.
Do I need to be enrolled in a university program throughout the duration of the competition?
All participating team members must remain full-time students in good standing at their university while participating in the competition.
Do I need to be a certain age?
Participants must be at or above the age of majority in the country, state, province or jurisdiction of residence at the time of entry.
Can I enroll if a family member is an Amazon employee?
Immediate family members and household members of Amazon employees, directors and contractors are not eligible to participate. See Official Rules for additional restrictions.
How many teams will be selected to participate?
All applications will be reviewed and evaluated by a panel of Amazon experts. Up to ten teams will be selected and sponsored by Amazon. All teams will receive a $250,000 grant intended to support two full-time students, a month of faculty time, free Alexa devices, and free AWS hosting including access to CPU and GPU based machines, SQL and NoSQL databases, and object storage. See Official Rules for details.
How many team members can our team have?
There are no minimum or maximum team member requirements. All team members must be enrolled in their university throughout the duration of the competition. All teams will receive a $250,000 grant regardless of how many members are on the team. We recommend a team with four to six students with diverse fields of study or areas of expertise.
Can students from different universities be on the same team?
Teams must be comprised of students attending the same university.
Can one university have more than one team?
Yes, universities may have more than one team.
Can I participate on two separate teams?
You can only be a part of one team for the duration of the competition.
Can undergraduate and graduate students work together?
Yes, teams may be comprised of undergraduate and graduate students.
Do I need a faculty advisor?
All teams must nominate a faculty advisor and include the faculty advisor’s consent in the applications.
What is the role of the faculty advisor?
Faculty advisors will advise students on technical directions and be a sounding board for new ideas, similar to a graduate school advisor. They will also act as the official representative from the university for this competition.
Can we add or remove team members during the competition?
During the competition, until April 29, 2020 faculty advisors may request to remove or add members to the team, subject to approval by Amazon.
Can we discuss our socialbot with faculty or students who aren’t on our team?
Only team members may work on their socialbots. However, the faculty advisor and other students and faculty members at your university may provide support and advice to your team and may co-author technical publications and research papers.
How do we apply?
Begin the application here.
What do we need to apply?
Once you have selected your team members, team leader, and faculty sponsor, you are ready to begin the application process. To see the questions, you can preview a pdf of the online application.
Do all team members have to apply?
Each team must have a team lead, who should apply on behalf of the whole team. Your application must include all of your team members’ information.
Is there an application fee?
There is no application fee.
How will teams be selected to participate?
All applications will be reviewed by a panel of Amazon employees. Teams will be selected based on the following criteria: (1) the potential scientific contribution to the field; (2) the technical merit of the approach; (3) the novelty of the idea; and (4) an assessment of the team’s ability to execute against their plan. Please be sure to provide enough detail in your application to enable our experts to evaluate your proposal.
What are the prizes for winning the competition?
A prize of $500,000 will be awarded to the team that creates the best socialbot. The second-place and third-place finalist teams will receive a $100,000 and a $50,000 prize, respectively. See the contest rules for details.
A prize of a $1 million research grant will be awarded to the winning team’s university if their socialbot achieves the grand challenge of conversing coherently and engagingly with humans for 20 minutes with a 4.0 or higher rating.
Do we get a stipend and devices to participate in the Alexa Prize?
Up to ten teams will be sponsored to participate in the Alexa Prize. These teams’ universities will receive a $250,000 research grant to fund the team members’ work over the year.
The sponsorship includes one Alexa-enabled device per team member for up to a total of three devices per team and one Alexa-enabled device per faculty advisor, free AWS services to support the development of their socialbot, and support from the Alexa team.
How can the grant be spent?
The grants will be awarded with the intention that they will support two full-time students for the duration of the competition and one month of the faculty advisor’s salary. No more than 35% of the research grants may be allocated to administrative fees. If your team would like to use the funds in another manner, your faculty advisor must receive approval from Amazon before doing so.
How will the prizes be distributed among a team?
The first, second, and third place prizes will be distributed equally among all registered team members. The official list of registered team members must be confirmed by April 29, 2020.
What are the key milestones of the competition?
Teams must submit their applications between March 4, 2019 and May 14, 2019. Between June 4 and June 7 2019, we will announce teams selected to participate. In the summer of 2019 teams will be invited to an Alexa Prize bootcamp at Amazon where they will receive training on the resources made available to all competing teams. The semifinals will be scheduled during March to April 2020. The final event of the competition will take place in May 2020.
If selected, when will we receive the stipend, devices, access to the alexa prize cobot toolkit, our aws account and be introduced to our point of contact?
We will reach out to all teams no later than June 7, 2019, with instructions on next steps. Up to ten teams will be selected to receive a $250,000 stipend, Alexa-enabled devices, free AWS services to support their development efforts, and support from the Alexa team.